Development Guides

Configurations

Configuration file

server:
    port: 9080
    addr: 0.0.0.0
    ## Allow models and namespaces loading from S3 buckets
    # resources:
    #   # s3 endpoint
    #   endpoint: rocky0:32082
    #   # s3 credentials
    #   user: minioadmin
    #   password: minioadmin
    #   # bucket where the models are stored
    #   modelsPath: models
    #   # bucket where the namespaces are store
    #   namespacesPath: namespaces
    #   # bucket where the policies are stored
    #   policiesPath: policies
    ## Default max number of entities returned by queries
    maxReturnedEntities: 1000
    # # Optional JWT algorithm, default algoritm is HS256
    # jwtAlgorithm: RS256
    # # Optional (if RS256)
    # # This config and the publicKey/privateKey are mutually exclusive, with priority given to jwksUri
    # jwksUri: http://keycloak-cluster-service.kosmos-iam.svc.cluster.local./realms/kosmos/
    # # Optional (if RS256)
    # # This config & privateKey are mutually exclusive with jwksUri, with priority given to jwksUri
    # publicKey: |-
    #   -----BEGIN PUBLIC KEY-----
    #   xxx
    #   -----END PUBLIC KEY-----
    # # Optional (if RS256)
    # # This config & publicKey are mutually exclusive with jwksUri, with priority given to jwksUri
    # privateKey: |-
    #   -----BEGIN PRIVATE KEY-----
    #   xxx
    #   -----END PRIVATE KEY-----
store:
    # use only one namespace if you don't know how to use it
    # namespace must be unique

    # default (TiDB) store example
    - namespace: test
      user: root
      password: 8k+m3bdVj5^*u@791X
      addr: 127.0.0.1:4000
          # readOnly: false # Default to 'false'
          # poolSize: 10 ## maximum pool size used when connecting to the store

          ## Postgres store example
          # user: postgres
          # password: postgres
          # addr: 127.0.0.1:5432
          # impl: postgresql
          # readOnly: false # Default to 'false'
          # poolSize: 10 ## maximum pool size used when connecting to the store

          ## Clickhouse store example
          # user: testuser
          # password: testpassword
          # addr: 127.0.0.1:9004
          # impl: clickhouse
          # readOnly: false # Default to 'false'
          # poolSize: 10 ## maximum pool size used when connecting to the store

          ## StarRocks store example
          # user: testuser
          # password: testpassword
          # addr: 127.0.0.1:9030
          # impl: starrocks
          # readOnly: false # Default to 'false'
        # poolSize: 10 ## maximum pool size used when connecting to the store

Token Validation

VStore uses JWT for user authentication and as such will try to validate them. This validation can be configured with the foolowing information :

The algorithm, which can be HS256 (default) or RS256
If using RS256 you must also provide either of :
- A jwksUri so that VStore may download the certificate necessary to validate JWT
- A publicKey to validate the JWT and optionally a privateKey to configure the VStore as an authority.
  - Do note that the privateKey field should only be used in development
- The jwksUri field has priority over the publicKey & privateKey fields. If it is filled, the keys will be ignored.

In the config file

server:
    [...]
    # Optional JWT algorithm, default algoritm is HS256
    jwtAlgorithm: RS256
    # Optional (if RS256)
    # This config and the publicKey/privateKey are mutually exclusive, with priority given to jwksUri
    jwksUri: http://keycloak-cluster-service.kosmos-iam.svc.cluster.local./realms/kosmos/
    # Optional (if RS256)
    # This config & privateKey are mutually exclusive with jwksUri, with priority given to jwksUri
    # publicKey: |-
    #   -----BEGIN PUBLIC KEY-----
    #   xxx
    #   -----END PUBLIC KEY-----
    # # Optional (if RS256)
    # # This config & publicKey are mutually exclusive with jwksUri, with priority given to jwksUri
    # privateKey: |-
    #   -----BEGIN PRIVATE KEY-----
    #   xxx
    #   -----END PRIVATE KEY-----
    [...]

Define a Namespace/Store

A Namespace/Store requires :

A database name which will be considered the namespace name
The endpoint & credentials to connect to the database

Optionally you can provide :

The store implementation, default to TiDB
The thread pool size to use to connect to the database, default to the available numbers of processors with a minimum or 1
Whether this VStore is a Read Only node, default to false
- A read only VStore will not fail on database initialization when it does not have enough permissions
Additional configuration properties to pass on to the data source

Stores/Namespaces may come either from the local configuration file or from a configuration file hosted on S3.

In the config file

server:
    # Bind port
    port: 9080
    # Bind address
    addr: 0.0.0.0
    # limit to the number of entity returned by a search request without override.
    maxReturnedEntities: 1000
store:
  # use only one namespace if you don't know how to use it
  # namespace must be unique
  - namespace: test
    # Information to reach the store
    user: root
    password: 8k+m3bdVj5^*u@791X
    addr: 127.0.0.1:4000
    # Store type, if not provided, default to 'tidb'
    impl: tidb
    readOnly: false # Default to 'false'
    # poolSize: 10 ## maximum pool size used when connecting to the store
    #properties:
      #my_additional_property: my_additional_value

    ## Postgres exemple
    # user: postgres
    # password: postgres
    # addr: 127.0.0.1:5432
    # impl: postgresql
    # readOnly: false # Default to 'false'
    # poolSize: 10 ## maximum pool size used when connecting to the store
    # properties:
      #my_additional_property: my_additional_value

    ## Clickhouse store exemple
    # user: testuser
    # password: testpassword
    # addr: 127.0.0.1:9004
    # impl: clickhouse
    # readOnly: false # Default to 'false'
    # poolSize: 10 ## maximum pool size used when connecting to the store
    # properties:
      #my_additional_property: my_additional_value

    ## StarRocks store example
    # user: testuser
    # password: testpassword
    # addr: 127.0.0.1:9030
    # impl: starrocks
    # readOnly: false # Default to 'false'
    # poolSize: 10 ## maximum pool size used when connecting to the store
    # properties:
      #my_additional_property: my_additional_value

From a s3 bucket

Alternatively you can store your namespaces files on a s3 server (only the store part will be taken into account). Multiple files are allowed.

Add your config (same as above) in your s3 bucket: mybucketns

Set the resources part in your config file.

server:
    port: 9080
    addr: 0.0.0.0
    resources:
        endpoint: kube0:31318
        user: minioadmin
        password: minioadmin
        # bucket where the model is store
        namespacesPath: mybucketns

Define the Policy

A policy is a set of rules that applies to a certain subsets of resources in a namespace. These rules may filter out or mask (hiding the values) from a query result. This is a namespaced resource.

In a policy the @data.colA anchor refers to the entity column colA whereas @identity.tags.security_attribute_1 refers to one of the suer security attribute.

Policies may come either from the local configuration file or from a configuration file hosted on S3.

In the config file

---
id: intel_policy
namespace: test
# # will only apply to the provided entity if defined
# entitiesScope: [person]
# # will be exclude for the provided entity if defined
# entitiesExclude: [equipment]
actions:
  # filter out false predicate

  # true if data cls field <= identity tag 'cls' value
  - type: filter
    predicate: "@data.cls <= @identity.tags.cls"

  # true if data stmts json fields fully contained in the json value of identity tag 'cls' value
  - type: filter
    predicate: "@data.stmts IN @identity.tags.stmts"

  # The following policy is kept commented as it clashes with the one that follows it. It only serves as reference.

  ## true if data 'thematics' json has a non null intersection with the 'thematics value of the macthing identity'
  ## eq: at least one element of identity thematics is inside thematics of data or thematics of data is null
  #- type: filter
  #  predicate: "@data.thematics IS NULL OR @identity.tags.thematics INTERSECT @data.thematics"

  # Equivalent to the above policy but with the 'masking' type
  # Mask the attributes if the predicate is false instead of preventing access
  # With the 'exclude' field you can choose which of the data's fields will NOT be masked
  - type: masking
    # If set will mask the data with the provided value, else null
    mask: M
    predicate: "@data.thematics IS NULL OR @identity.tags.thematics INTERSECT @data.thematics"
    # field to exclude from the masking process (-> will not be masked)
    exclude:
      - id
      - age

From a s3 bucket

Alternatively you can store your policies files on a s3 server. Multiple files are allowed.

Add your policy (same as above) in your s3 bucket: mybucketpol

Set the resources part in your config file.

server:
  port: 9080
  addr: 0.0.0.0
  resources:
    endpoint: kube0:31318
    user: minioadmin
    password: minioadmin
    # bucket where the model is store
    policiesPath: mybucketpol
...

Define the identities

An identity is a user representation :

with a unique pair of id and tenantId
with a set of security tags that will be used in concordance with policies to define which query result they are able to see unfiltered/unmasked.

A user may be set as admin to avoid all policies by setting the profile key as admin.

Additionally, an admin may be set as a multi-tenant admin (superadmin) by setting its tenantId with the value *.

Identities may come either from the configuration file or retrieved dynamically from a VStore.

In the config file

---
id: myuser
tenantId: kosmos
tags:
  - key: cls
    value: TOP_SECRET
  - key: stmts
    value: '{"SF":1,"ACSSI":1,"ECLAIR":1}'
  - key: thematics
    value: "NOTIN,PROLI"
---
id: myadmin
profile: admin
tenantId: 'kosmos'
---
id: mysuperadmin
profile: admin
tenantId: '*'

Dynamic Identities

It is possible to provide identities dynamically from another vstore. To do so you will need to configure the identities part:

server:
  port: 9080
  addr: 0.0.0.0
  maxReturnedEntities: 1000
  [...]
  identities:
    # host to reach (can be another vstore)
    host: 127.0.0.1
    uri: /v0/test/entities/identity
    conditionalFilter: tenant_id = 'athea'
    # polling delay (default 3 sec)
    # delayMs: 5000
    # bearer token to use (user will need to have full access to all identities, admin, etc...)
    #pageSize: 5000 # number of user to return per request (if you have more users, multiple request will be made with pagination)
    token: |-
      [...]
  ...

The conditional filter will be added to the usual request and allow you to choose which user to retrieve from the identities source.

For the vstore owner of entities you will need the following model:

entities:
  - id: identity
    namespace: test
    primaryKeys:
      - id
    fields:
      - name: id
        type: VARCHAR(64)
      # The 'tags' should be a json array which contains the tags found in your identity in Json objet with the keys 'key' & 'value'
      # eg: if you only have a tag cls which is of type "ENUM('NP', 'DR', 'SECRET', 'TOP_SECRET')"
      #     an example of tags would be '[{"key":"cls","value":"TOP_SECRET"}]'
      - name: tags
        type: VARCHAR(256)
      # set on insert and update with millisec timestamp
      # for now this will be used to detect modification in the polling phase.
      - name: ts
        type: BIGINT
      - name: tenantId
        type: VARCHAR(64)

Define a model

A model is a series of entities, each of which has fields. This is a namespaced resource.

Models may come either from the local configuration file or from a configuration file hosted on S3.

In the config file

---
version: 1
entities:
  - id: person

    # The namespace the entity belong to
    # use only one for all your entities if you don't know how to use it
    namespace: test

    ## will clear entity (use it only if you know what you are doing)
    # markForDeletion: true

    # Activate entity modification history
    # Only works on entity with single field primary key
    history: true

    # Optional list of mutations to apply to an existing entity to match itw new model
    # This field is used to provide hints on how the previous model should be modified
    # to match the new one.
    # For more information see the 'Mutate a model' section
    mutations:
      add:
          indexes: []
          fields: []
          constraints: []
      delete:
          indexes: []
          fields: []
          constraints: []

    # Optional indexes list
    # use only for high cardinality values and frequents queries on it.
    indexes:
      - name: testidx
        expression: (testidx)
    # Provide the list of field that will serve as a primary key for your entity
    # Currently history mode accept nly table with a single field primary key
    primaryKeys:
      - id
    # Optional constraints list (Primary key excluded)
    # Some limitations are present and depends on the backing database.
    # Check the 'Adding constraints to the model' section for more information
    constraints:
      - expression: UNIQUE(surname)
        name: surname
      - expression: CHECK(age>21)
        name: check_1
    fields:
      - name: id
        type: VARCHAR(128)
      - name: firstName
        type: VARCHAR(128)
      - name: lastName
        type: VARCHAR(128)
      - name: gender
        type: VARCHAR(128)
      - name: age
        type: SMALLINT
      - name: surname
        type: VARCHAR(128)
      - name: ref
        type: VARCHAR(128)
      - name: cls
        security: true
        # Hierarchical sort
        type: ENUM('NP', 'DR', 'SECRET', 'TOP_SECRET')
      - name: stmts
        type: JSON
        security: true
      - name: testidx
        type: VARCHAR(128)
      - name: ts
        type: BIGINT
      - name: thematics
        type: SET('PROLI','TARGET','NUCLEAR')
        security: true
      - name: testaddfield
        type: VARCHAR(128)
      - name: testbigjson
        type: JSON
      - name: testdouble
        type: DOUBLE
      - name: testfloat
        type: FLOAT

Adding constraints to the model

The constraints field should not be used to create PRIMARY KEY constraints, the primaryKeys field should be used instead !

The contstraints field implementations has limitation depending on which database is backing your namespace :

TiDB/PostgreSQL : the [NOT] NULL constraint should be kept in the fields section
Clickhouse : no support for constraints field
StarRocks :
- Only the UNIQUE & FOREIGN KEY constraint are supported
- UNIQUE :
  - The name of a UNIQUE constraint MUST BE unique_constraints
  - Only a single UNIQUE constraint may exist at a time. If multiple are declared/added through mutation, only the last one will remain.
- FOREIGN KEY :
  - The name of a FOREIGN KEY constraint MUST BE foreign_key_constraints
  - Only a single FOREIGN KEY constraint may exist at a time. If multiple are declared/added through mutation, only the last one will remain.

Mutate a model that already exists

Model mutation allow to add or delete fields and indexes but not to change their name, type or constraints.

When you want to change your model from a version n to n+1 you need to :

Update the version field to n+1.
Change the fields & indexes fields to match you new model.
Fill the mutations field with all the actions necessary to change the entity from version n to version n+1.

Note: Deleting and adding the same field/index will not cancel each other. A deletion followed by a recreation will happen.

As an exemple we will consider that the following model has been used to create an entity previously :

---
version: 1
entities:
  - id: person
    namespace: test
    history: true
    indexes:
      - name: firstName
        expression: (firstName)
    primaryKeys:
      - id
    constraints:
        - expression: UNIQUE(surname)
          name: surname_is_unique
    fields:
      - name: id
        type: VARCHAR(128)
      - name: firstName
        type: VARCHAR(128)
      - name: surname
        type: VARCHAR(128)
      - name: testdouble
        type: DOUBLE
      - name: cls
        security: true
        # Hierarchical sort
        type: ENUM('NP', 'DR', 'SECRET', 'TOP_SECRET')

Let's say we want to :

remove the field 'testdouble'
remove the index 'firstName'
remove the constraint 'surname_is_unique'
add a new field 'lastName' of type 'VARCHAR(128)' & a new field 'age' of type 'SMALLINT'
add a new index 'lastName' on the new field 'lastName'
add a new constraint that state 'age' should be greater than 21

Our new model would look like this :

---
version: 2
entities:
  - id: person
    namespace: test
    history: true
    mutations:
      add:
        indexes:
          - name: lastName
            expression: (lastName)
        fields:
          - name: lastName
            type: VARCHAR(128)
        constraints:
          - expression: CHECK(age>21)
            name: check_age
      delete:
        indexes:
          - name: firstName
        fields:
          - name: testdouble
        constraints:
            - name: surname_is_unique
    indexes: # firstName should no longer exists, as such only the lastName index is present in here
      - name: lastName
        expression: (lastName)
    primaryKeys:
      - id
    fields:
      - name: id
        type: VARCHAR(128)
      - name: firstName
        type: VARCHAR(128)
      - name: cls
        security: true
        # Hierarchical sort
        type: ENUM('NP', 'DR', 'SECRET', 'TOP_SECRET')
      - name: lastName # testdouble has been removed and lastName has taken its place
        type: VARCHAR(128)

From a s3 bucket

Alternatively you can store your model files on a s3 server. Multiple files are allowed.

Add your model (with the same constraints and possibilities as above) in your s3 bucket, eg : mybucketmodel

Set the resources part in your config file.

server:
  port: 9080
  addr: 0.0.0.0
  resources:
    endpoint: kube0:31318
    user: minioadmin
    password: minioadmin
    # bucket where the model is stored
    modelsPath: mybucketmodel
[...]

Use the Change Data Capture (CDC) feature

CDC format is a lightweight version of the Canal JSON CDC format.

Exemple

Modify entity when history is activated:

HTTP PUT http://127.0.0.1:9080/v0/test/entities/person/937dd176-3548-4ddb-a3f3-a2b180286322
{
    "age" : 98
}

Read CDC entity

HTTP GET http://127.0.0.1:9080/v0/test/entities/person_history?filter=ref = '937dd176-3548-4ddb-a3f3-a2b180286322'

Return :

{
  "values" : [ {
    "ref" : "937dd176-3548-4ddb-a3f3-a2b180286322",
    "ts" : 1713873822105,
    "type" : "UPDATE",
    "identity" : "eric",
    "data" : "{\"age\": 98}",
    "old" : "{\"age\": 1}",
    "stmts" : "{\"SF\": 1, \"ACSSI\": 1, \"ECLAIR\": 1}",
    "thematics" : "PROLI,TARGET,NUCLEAR",
    "cls" : "DR"
  } ]
}

Dynamic Dataset/Views

Dynamic dataset/views allow views creation with policy for database direct access with native driver.

You can either provide a BODY which contains a user definition (reserved for admins) OR a null body, in which case the user you are using to make the request will be used instead.

Exemples

Admin API : Create Dataset with arbitrary tags

The following will create views based on provided model with the following format:

<entity_id_from_model>__<identity.id>__<identity.tenantId>

with the provided sample identity (see HTTP POST request):

<entity_id_from_model>__vuser__athea

# request done by creating a user (reserved for admins)
HTTP POST http://127.0.0.1:9080/v0/<namespace>/vdataset
{
    "id": "vuser",
    "tenantId": "athea",
    "tags": [
      {
        "key": "cls",
        "value": "TOP_SECRET"
      },
      {
        "key": "stmts",
        "value": ["SF","ACSSI","ECLAIR"]
      },
      {
        "key": "thematics",
        "value": ["NOTIN","PROLI"]
      }
    ]
}

Returns :

{
    "status": 0,
    "token": "nY*66nNVcaN2",
    "identity": "vuser__athea",
    "vdatasets":[
        "person_history__vuser__athea",
        "person__vuser__athea"
    ]
}

The service account vuser__athea and token nY*66nNVcaN2 are now available for generic postgresql or mysql (TiDB/Clickhouse/StarRocks) driver usage.

Create Dataset with implicit tags

The following will create views based on provided model with the following format:

<entity_id_from_model>__<identity.id>__<identity.tenantId>

with the provided sample identity (see HTTP POST request):

<entity_id_from_model>__vuser__athea

We consider that the calling user has the id realuser and tenant athea

# request done with a null body, the permissions are thus of the calling user
HTTP POST http://127.0.0.1:9080/v0/<namespace>/vdataset

Returns :

{
    "status": 0,
    "token": "qL*0hd0r*ssL",
    "identity": "realuser__athea",
    "vdatasets":[
        "person_history__realuser__athea",
        "person__realuser__athea"
    ]
}

The service account realuser__athea and token qL*0hd0r*ssL are now available for generic postgresql or mysql (TiDB/Clickhouse/StarRocks) driver usage.

vStore API

Authentication

The authentication should be made with the Authorization header with a bearer token. This bearer token should contains these fields :

preferred_username
tenant_id

In order for the authorization to succeed, this pair should match a pair or id & tenantId in the entities known to vstore.

Request models

`GET /v0/models`

There are no parameters for this request.

Exemple

HTTP GET http://127.0.0.1:9080/v0/models

Return :

{
  "version" : 0,
  "entities" : [ {
    "id" : "person",
    "namespace" : "test",
    "history" : true,
    "mutation" : null,
    "indexes" : [ {
      "name" : "testidx",
      "expression" : "(testidx)"
    } ],
    "markForDeletion" : false,
    "fields" : [ {
      "name" : "id",
      "newName" : null,
      "type" : "VARCHAR(128) PRIMARY KEY",
      "security" : false
    }, {
      "name" : "firstName",
      "newName" : null,
      "type" : "VARCHAR(128)",
      "security" : false
    }, {
      "name" : "lastName",
      "newName" : null,
      "type" : "VARCHAR(128)",
      "security" : false
    }, {
      "name" : "gender",
      "newName" : null,
      "type" : "VARCHAR(128)",
      "security" : false
    }, {
      "name" : "age",
      "newName" : null,
      "type" : "SMALLINT",
      "security" : false
    }, {
      "name" : "surname",
      "newName" : null,
      "type" : "VARCHAR(128)",
      "security" : false
    }, {
      "name" : "ref",
      "newName" : null,
      "type" : "VARCHAR(128)",
      "security" : false
    }, {
      "name" : "cls",
      "newName" : null,
      "type" : "ENUM('NP', 'DR', 'SECRET', 'TOP_SECRET')",
      "security" : true
    }, {
      "name" : "stmts",
      "newName" : null,
      "type" : "JSON",
      "security" : true
    }, {
      "name" : "testidx",
      "newName" : null,
      "type" : "VARCHAR(128)",
      "security" : false
    }, {
      "name" : "ts",
      "newName" : null,
      "type" : "BIGINT",
      "security" : false
    }, {
      "name" : "thematics",
      "newName" : null,
      "type" : "SET('PROLI','TARGET','NUCLEAR')",
      "security" : true
    }, {
      "name" : "testaddfield",
      "newName" : null,
      "type" : "VARCHAR(128)",
      "security" : false
    }, {
      "name" : "testbigjson",
      "newName" : null,
      "type" : "JSON",
      "security" : false
    }, {
      "name" : "testdouble",
      "newName" : null,
      "type" : "DOUBLE",
      "security" : false
    }, {
      "name" : "testfloat",
      "newName" : null,
      "type" : "FLOAT",
      "security" : false
    }, {
      "name" : "count",
      "newName" : null,
      "type" : "BIGINT",
      "security" : false
    } ]
  } ]
}

Create element(s) in an entity

`POST /v0/<namespace>/entities/<entity>`

Path parameter	Description
`namespace`	Namespace of the entity in which you want to create elements
`entity`	Name of the entity in which you want to create elements

Exemple

HTTP POST http://127.0.0.1:9080/v0/test/entities/person
{
  "values": [
    {
      "id": "10009c11-ffb5-4bce-9806-fa4bffb22fb5",
      "firstName": "John10000",
      "cls": "SECRET",
      "count": 999
    },
    {
      "id": "10009c11-ffb5-4bce-9806-fa4bffb22fb6",
      "firstName": "John10001",
      "cls": "DR",
      "count": 1000
    }
  ]
}

Query element(s) of an entity

`GET /v0/<namespace>/entities/<entity>`

Path parameter	Description
`namespace`	Namespace of the entity in which you want to apply the SQL query
`entity`	Name of the entity in which you want to apply the SQL query

Query Parameter	Description
`actionstype`	Allow additional configuration on policies. The only supported is currently `masking=filter` which remove element with masked fields
`filter`	SQL like condition that should be used to limit the element that will be returned
`limit`	Maximum number of result to returns. Mutually exclusive with the body parameter of the same name
`offset`	Number of result that should be skipped in the initial result. Mutually exclusive with the body parameter of the same name
`sort`	SQL compliant mode of ordering results. Mutually exclusive with the body parameter of the same name

Body Parameter	Description
`limit`	Maximum number of result to returns. Mutually exclusive with the query parameter of the same name
`offset`	Number of result that should be skipped in the initial result. Mutually exclusive with the query parameter of the same name
`sort`	SQL compliant mode of ordering results. Mutually exclusive with the query parameter of the same name

The actionstype, limit, offset & sort parameters are optional.

`GET /v0/<namespace>/entities/<entity>/_query`

Path parameter	Description
`namespace`	Namespace of the entity in which you want to apply the SQL query
`entity`	Name of the entity in which you want to apply the SQL query

Query Parameter	Description
`actionstype`	Allow additional configuration on policies. The only supported is currently `masking=filter` which remove element with masked fields
`limit`	Maximum number of result to returns. Mutually exclusive with the body parameter of the same name
`offset`	Number of result that should be skipped in the initial result. Mutually exclusive with the body parameter of the same name
`orderby`	SQL compliant mode of ordering results. Mutually exclusive with the body parameter of the same name

Body Parameter	Description
`filter`	SQL like condition that should be used to limit the element that will be returned
`limit`	Maximum number of result to returns. Mutually exclusive with the query parameter of the same name
`offset`	Number of result that should be skipped in the initial result. Mutually exclusive with the query parameter of the same name
`orderby`	SQL compliant mode of ordering results. Mutually exclusive with the query parameter of the same name

The actionstype, limit, offset & orderby parameters are optional.

Exemple

Get all person with id = 5657de02-f9c9-448c-baa4-29a77ee2d540:

HTTP GET http://127.0.0.1:9080/v0/test/entities/person?filter=id = '5657de02-f9c9-448c-baa4-29a77ee2d540'

Same with a limit, offset and sort clause:

HTTP GET http://127.0.0.1:9080/v0/test/entities/person?filter=id = '5657de02-f9c9-448c-baa4-29a77ee2d540'&limit=10&offset=10&orderby=count asc

Or with a POST for too long url query:

HTTP POST http://127.0.0.1:9080/v0/test/entities/person/_query
{
  "filter": "id = '5657de02-f9c9-448c-baa4-29a77ee2d540'"
}

Same with a limit, offset and sort clause:

HTTP POST http://127.0.0.1:9080/v0/test/entities/person/_query
{
  "filter": "id = '5657de02-f9c9-448c-baa4-29a77ee2d540'",
  "limit": 10,
  "offset": 10,
  "orderby": "count asc"
}

Get all person with firstName = John4203:

HTTP GET http://127.0.0.1:9080/v0/test/entities/person?filter=firstName = 'John4203'

Return :

{
  "values" : [ {
    "id" : "1ddbf8b0-6065-4b56-a2fc-e1be3e21c3f2",
    "firstName" : "John4203",
    "lastName" : "Doe4203",
    "gender" : "F",
    "age" : 30,
    "surname" : "James5657de02-f9c9-448c-baa4-29a77ee2d540",
    "ref" : "5657de02-f9c9-448c-baa4-29a77ee2d540link",
    "cls" : "TOP_SECRET",
    "stmts" : "{\"ACSSI\": 1, \"ECLAIR\": 1, \"SF\": 1}",
    "thematics" : "{\"NUCLEAR\": 1, \"PROLI\": 1, \"TARGET\": 1}"
  } ]
}

If you need to transform on demand an actions type (see policies config) from masking to filter, use:

HTTP GET http://127.0.0.1:9080/v0/test/entities/person?actionstype=masking=filter

Update element(s) of an entity

By primary key : `PUT /v0/<namespace>/entities/<entity>/<id>`

Path Parameter	Description
`namespace`	Namespace of the entity in which you want to update an element
`entity`	Name of the entity in which you want to update an element
`id`	Primary key of the element you wish to update

You must provide a JSON body with the fields you want to update and their new value.

Exemple

Update the field age of a person with id = 1ddbf8b0-6065-4b56-a2fc-e1be3e21c3f2 (id must be declared as a primary key):

HTTP PUT http://127.0.0.1:9080/v0/test/entities/person/1ddbf8b0-6065-4b56-a2fc-e1be3e21c3f2
{
    "age" : 34
}

Return (HTTP 200, status: 1 mean one element modified):

{
    "status": 1,
    "msg": null
}

By filter : `PUT /v0/<namespace>/entities/<entity>`

Updating by filter allow to provide a filter which will allow to select the element which will be updated. (Filtering through security attributes still applies) You should filter using query parameter as the preferred option and use the filter through body parameter only if your filter gets too long.

Filtering with query parameter

Path Parameter	Description
`namespace`	Namespace of the entity in which you want to update an element
`entity`	Name of the entity in which you want to update an element

Query Parameter	Description
`filter`	SQL like condition that should be used to limit the element that should receive the update

You must provide a JSON body with the fields you want to update and their new value.

Filtering with body parameter

Path Parameter	Description
`namespace`	Namespace of the entity in which you want to update an element
`entity`	Name of the entity in which you want to update an element

Body Parameter	Description
`filter`	SQL like condition that should be used to limit the element that should be updated
`update`	JSON object with the fields you want to update and their new value.

Exemple

HTTP PUT http://127.0.0.1:9080/v0/test/entities/person?filter= id = '1ddbf8b0-6065-4b56-a2fc-e1be3e21c3f2'
{
    "age" : 34
}

HTTP PUT http://127.0.0.1:9080/v0/test/entities/person
{
    "filter" : "id = '1ddbf8b0-6065-4b56-a2fc-e1be3e21c3f2'",
    "update": {
        "age" : 34
    }
}

Both return (HTTP 200, status: 1 mean one element modified):

{
    "status": 1,
    "msg": null
}

Delete element(s) of an entity

By primary key : `PUT /v0/<namespace>/entities/<entity>/<id>`

Path Parameter	Description
`namespace`	Namespace of the entity in which you want to delete an element
`entity`	Name of the entity in which you want to delete an element
`id`	Primary key of the element you wish to delete

Exemple

Delete a person with id = 1ddbf8b0-6065-4b56-a2fc-e1be3e21c3f2 (id must be declared as a primary key):

HTTP DELETE http://127.0.0.1:9080/v0/test/entities/person/1ddbf8b0-6065-4b56-a2fc-e1be3e21c3f2

Return (HTTP 204, status: 1 mean one element deleted):

{
    "status": 1,
    "msg": null
}

By filter : `DELETE /v0/<namespace>/entities/<entity>`

Deleting by filter allow to provide a filter which will allow to select the element which will be deleted. (Filtering through security attributes still applies) You should filter using query parameter as the preferred option and use the filter through body parameter only if your filter gets too long.

Filtering with query parameter

Path Parameter	Description
`namespace`	Namespace of the entity in which you want to delete an element
`entity`	Name of the entity in which you want to delete an element

Query Parameter	Description
`filter`	SQL like condition that should be used to limit the element that should be deleted

Filtering with body parameter

Path Parameter	Description
`namespace`	Namespace of the entity in which you want to delete an element
`entity`	Name of the entity in which you want to delete an element

Body Parameter	Description
`filter`	SQL like condition that should be used to limit the element that should be deleted

Exemple

HTTP DELETE http://127.0.0.1:9080/v0/test/entities/person?filter= id = '1ddbf8b0-6065-4b56-a2fc-e1be3e21c3f2'

HTTP PUT http://127.0.0.1:9080/v0/test/entities/person
{
    "filter" : "id = '1ddbf8b0-6065-4b56-a2fc-e1be3e21c3f2'"
}

Both return (HTTP 204, status: 1 mean one element deleted):

{
    "status": 1,
    "msg": null
}

Get an entity history

`GET /v0/<namespace>/entities/<entity>_history`

Path Parameter	Description
`namespace`	Namespace of the entity in which you want to update an element
`entity`	Name of the entity in which you want to update an element

Query Parameter	Description
`filter`	SQL like condition that should be used to limit the element that should be deleted

Do note that the history tables have predetermined fields, the ref field for exemple being the unique ID of the original element concerned by the history element.

Exemple

HTTP GET http://127.0.0.1:9080/v0/test/entities/person_history?filter=ref = '1ddbf8b0-6065-4b56-a2fc-e1be3e21c3f2'

Return:

{
    "values" : [ {
        "ref" : "1ddbf8b0-6065-4b56-a2fc-e1be3e21c3f2",
        "ts" : 1713873822105,
        "type" : "UPDATE",
        "identity" : "eric",
        "data" : "{\"age\": 34}",
        "old" : "{\"age\": 32}",
        "stmts" : "{\"SF\": 1, \"ACSSI\": 1, \"ECLAIR\": 1}",
        "thematics" : "PROLI,TARGET,NUCLEAR",
        "cls" : "DR"
    } ]
}

Use the DDL api

The DDL api allow the use of SQL on an entity.

`POST /v0/<namespace>/entities/<entity>/_ddl`

Path parameter	Description
`namespace`	Namespace of the entity in which you want to apply the SQL query
`entity`	Name of the entity in which you want to apply the SQL query

Body Parameter	Description
`alter`	SQL action that end with a `;`

Exemple

Add a column

POST http://127.0.0.1:9080/v0/iam/entities/person/_ddl
{
  "alter": "ADD COLUMN phone VARCHAR(128);"
}

Rename a column

POST http://127.0.0.1:9080/v0/iam/entities/person/_ddl
{
  "alter": "RENAME COLUMN phone TO phone2;"
}

Drop a column

POST http://127.0.0.1:9080/v0/iam/entities/person/_ddl
{
  "alter": "DROP COLUMN phone2;"
}

Create a vDataset

`POST /v0/<namespace>/vdataset`

If you are a normal user, only the path parameter can be provided, the vUser will be created based on the calling user.

Path parameter	Description
`namespace`	Namespace of the entity in which you want to create the vDatasets

If you are an admin user, you can provide the following fields to create a vUser with arbitrary attributes.

Body Parameter	Description
`id`	User id
`tenantId`	Tenant of the user
`tags`	Security tags of the user

Exemple

HTTP POST http://127.0.0.1:9080/v0/<namespace>/vdataset
{
    "id": "vuser",
    "tenantId": "athea",
    "tags": [
      {
        "key": "cls",
        "value": "TOP_SECRET"
      },
      {
        "key": "stmts",
        "value": ["SF","ACSSI","ECLAIR"]
      },
      {
        "key": "thematics",
        "value": ["NOTIN","PROLI"]
      }
    ]
}

HTTP POST http://127.0.0.1:9080/v0/<namespace>/vdataset

vStore Operator API

The current API is to be used inside a kubernetes cluster and as such does not use TLS.

Authentication

You need to provide your vStore token the same was as if you were querying vStore itself, as a bearer token in a Authorization header.

Without this token, or an invalid token you will receive a http 401 error.

Get vStore readiness for a given model version

# Request
GET /vstore/model/status?version=<version_number>

# Response
{"status":"<status>"}

The version_number you provide must match the version returned by the GET /v0/models to consider the vStore that answered as Ready.

The three possible status are :

NotReady : if 0 replica have the desired version
InProgress : if 1 or more replica, but no all of them, have teh desired version
Ready : if all the replicas have the desired version

remarque

When a vStore pod is unable to answer (eg: restarting) it is considered as not ready

Configurations​

Configuration file​

Token Validation​

In the config file​

Define a Namespace/Store​

In the config file​

From a s3 bucket​

Define the Policy​

In the config file​

From a s3 bucket​

Define the identities​

In the config file​

Dynamic Identities​

Define a model​

In the config file​

Adding constraints to the model​

Mutate a model that already exists​

From a s3 bucket​

Use the Change Data Capture (CDC) feature​

Exemple​

Dynamic Dataset/Views​

Exemples​

Admin API : Create Dataset with arbitrary tags​

Create Dataset with implicit tags​

vStore API​

Authentication​

Request models​

GET /v0/models​

Exemple​

Create element(s) in an entity​

POST /v0/<namespace>/entities/<entity>​

Exemple​

Query element(s) of an entity​

GET /v0/<namespace>/entities/<entity>​

GET /v0/<namespace>/entities/<entity>/_query​

Exemple​

Update element(s) of an entity​

By primary key : PUT /v0/<namespace>/entities/<entity>/<id>​

Exemple​

By filter : PUT /v0/<namespace>/entities/<entity>​

Filtering with query parameter​

Filtering with body parameter​

Exemple​

Delete element(s) of an entity​

By primary key : PUT /v0/<namespace>/entities/<entity>/<id>​

Exemple​

By filter : DELETE /v0/<namespace>/entities/<entity>​

Filtering with query parameter​

Filtering with body parameter​

Exemple​

Get an entity history​

GET /v0/<namespace>/entities/<entity>_history​

Exemple​

Use the DDL api​

POST /v0/<namespace>/entities/<entity>/_ddl​

Exemple​

Add a column​

Rename a column​

Drop a column​

Create a vDataset​

POST /v0/<namespace>/vdataset​

Exemple​

vStore Operator API​

Authentication​

Get vStore readiness for a given model version​

Configurations

Configuration file

Token Validation

In the config file

Define a Namespace/Store

In the config file

From a s3 bucket

Define the Policy

In the config file

From a s3 bucket

Define the identities

In the config file

Dynamic Identities

Define a model

In the config file

Adding constraints to the model

Mutate a model that already exists

From a s3 bucket

Use the Change Data Capture (CDC) feature

Exemple

Dynamic Dataset/Views

Exemples

Admin API : Create Dataset with arbitrary tags

Create Dataset with implicit tags

vStore API

Authentication

Request models

`GET /v0/models`

Exemple

Create element(s) in an entity

`POST /v0/<namespace>/entities/<entity>`

Exemple

Query element(s) of an entity

`GET /v0/<namespace>/entities/<entity>`

`GET /v0/<namespace>/entities/<entity>/_query`

Exemple

Update element(s) of an entity

By primary key : `PUT /v0/<namespace>/entities/<entity>/<id>`

Exemple

By filter : `PUT /v0/<namespace>/entities/<entity>`

Filtering with query parameter

Filtering with body parameter

Exemple

Delete element(s) of an entity

By primary key : `PUT /v0/<namespace>/entities/<entity>/<id>`

Exemple

By filter : `DELETE /v0/<namespace>/entities/<entity>`

Filtering with query parameter

Filtering with body parameter

Exemple

Get an entity history

`GET /v0/<namespace>/entities/<entity>_history`

Exemple

Use the DDL api

`POST /v0/<namespace>/entities/<entity>/_ddl`

Exemple

Add a column

Rename a column

Drop a column

Create a vDataset

`POST /v0/<namespace>/vdataset`

Exemple

vStore Operator API

Authentication

Get vStore readiness for a given model version