On this page

    Models

    When submitting any of these models to the catalog REST API, you must specify the entityType property with the correct value.

    The Enterprise Edition includes additional properties per catalog entity, which are described here.

    CatalogEntitySummary

    A summary representation of a catalog entity, used when entities are listed as children.

    {
      "id": String [immutable, generated by Dremio],
      "path": [String] [immutable, generated by Dremio],
      "tag": String [immutable, generated by Dremio],
      "type": String [DATASET, CONTAINER, FILE],
      "datasetType": [VIRTUAL, PROMOTED, DIRECT] [immutable, optional],
      "containerType": [SPACE, SOURCE, FOLDER, HOME] [immutable, optional]
    }
    

    Fields

    NameTypeDescription
    idStringCatalog Entity ID. Generated by Dremio, immutable.
    path[String]Catalog Entity path. Immutable.
    tagStringIdentifies the instance of the entity, changed each time it is modified. Generated by Dremio, immutable.
    typeStringThe entity type, must be either DATASET, CONTAINER or FILE. Generated by Dremio, immutable.
    datasetTypeStringThe dataset type, must be either VIRTUAL, PROMOTED or DIRECT. Only present if type is dataset. Generated by Dremio, immutable.
    containerTypeStringThe container type, must be either SPACE, SOURCE, FOLDER or HOME. Only present if type is containerType. Generated by Dremio, immutable.

    Dataset

    Represents a dataset in Dremio. All datasets returned by the REST API will have an entityType of dataset.

    The JSON representation of a dataset looks like this:

    {
      "entityType": "dataset", [immutable, generated by Dremio]
      "id": String [immutable, generated by Dremio],
      "path": [String] [immutable after creation],
      "tag": String [immutable, generated by Dremio],
      "type": String ["PHYSICAL_DATASET", "VIRTUAL_DATASET"] [immutable],
      "fields": [DatasetField] [immutable],
      "createdAt": String (RFC3339 date) [immutable, generated by Dremio],
      "accelerationRefreshPolicy": DatasetAccelerationRefreshPolicy [optional, only for physical datasets in a source],
      "sql": String [optional, required for virtual datasets],
      "sqlContext": [String] [optional, only for virtual datasets],
      "format": DatasetFormat [optional, required for promoted datasets],
      "approximateStatisticsAllowed": Boolean [optional, introduced in Dremio 2.1.0]
    }
    

    Fields

    NameTypeDescription
    idStringDataset ID. Generated by Dremio, immutable.
    path[String]Dataset path. Immutable after creation.
    tagStringIdentifies the instance, changed each time it is modified. Generated by Dremio, immutable.
    typeStringThe dataset type, must be either PHYSICAL_DATASET or VIRTUAL_DATASET. Immutable after creation.
    fields[FieldSchema]The dataset fields representing the schema of the dataset. Immutable.
    createdAtStringRFC3339 date (example: 2017-10-27T21:08:22.858Z) representing the creation datetime. Immutable.
    accelerationRefreshPolicyDatasetAccelerationRefreshPolicyRepresents the acceleration refresh policy for the dataset. Applies only to physical datasets that exist in a source.
    sqlStringThe sql for the dataset, applies only to virtual datasets and required for them.
    sqlContext[String]The context for the sql, applies only to virtual datasets and is optional.
    formatDatasetFormatThe dataset format configuration, applies only to promoted physical datasets and is required.
    approximateStatisticsAllowedBooleanWhen set, count distinct queries will return approximate results.

    FieldSchema

    Represents a field’s schema in Dremio.

    The JSON representation of a field looks like this:

    {
      "name": String - the field name,
      "type": {
        "name": String ["STRUCT", "LIST", "UNION", "INTEGER", "BIGINT", "FLOAT", "DOUBLE", "VARCHAR", "VARBINARY", "BOOLEAN", "DECIMAL", "TIME", "DATE", "TIMESTAMP", "INTERVAL DAY TO SECOND", "INTERVAL YEAR TO MONTH"],
        "subSchema": [DatasetField] [optional],
        "precision": Number [optional],
        "scale": Number [optional]
      }
    }
    

    For complex types (LIST, STRUCT, UNION), subSchema will provide a list of DatasetField representing the composition. For example, UNION will have a subSchema which represents all the primitive types that have been detected.

    For DECIMAL type, precision/scale are provided.

    DatasetAccelerationRefreshPolicy

    Represents the acceleration refresh policy for a dataset.

    {
      "refreshPeriodMs": Number,
      "gracePeriodMs": Number,
      "method": String ["FULL", "INCREMENTAL"],
      "refreshField": String [optional]
    }
    

    Fields

    NameTypeDescription
    refreshPeriodMsNumberHow often (in milliseconds) to refresh all reflections on the dataset.
    gracePeriodMsNumberHow old (in milliseconds) data in a reflection can be and still be used for accelerating queries.
    methodStringFor every refresh, either a FULL or an INCREMENTAL (only works in certain cases, please see docs) update.
    refreshFieldStringFor certain datasets, a refreshField can be set if method is INCREMENTAL.

    DatasetFormat

    Folders/files can be promoted to physical datasets by applying a format. When applying a format to a folder, all files in that folder should conform to the selected type.

    Text (delimited)

    Applies to text files with delimiters (CSV, TSV, etc).

    {
      "type": "Text",
      "fieldDelimiter": String,
      "lineDelimiter": String,
      "quote": String,
      "comment": String,
      "escape": String,
      "skipFirstLine": Boolean,
      "extractHeader": Boolean,
      "trimHeader": Boolean,
      "autoGenerateColumnNames": Boolean
    }
    

    JSON

    {
      "type": "JSON"
    }
    

    Parquet

    {
      "type": "Parquet"
    }
    

    Excel

    {
      "type": "Excel",
      "sheetName": String,
      "extractHeader": Boolean,
      "hasMergedCells": Boolean
    }
    

    XLS

    {
      "type": "XLS",
      "sheetName": String,
      "extractHeader": Boolean,
      "hasMergedCells": Boolean
    }
    

    AVRO

    {
      "type": "Avro"
    }
    

    Space

    Represents a space in Dremio. All spaces returned by the REST API will have an entityType of space.

    The JSON representation of a space looks like this:

    {
      "entityType": "space" [immutable, generated by Dremio],
      "id": String [immutable, generated by Dremio],
      "name": String [immutable after creation],
      "tag": String [immutable, generated by Dremio],
      "createdAt": String (RFC3339 date) [immutable],
      "children": [CatalogEntitySummary] [optional] [immutable, generated by Dremio]
    }
    

    Fields

    NameTypeDescription
    idStringSource ID. Generated by Dremio, immutable.
    nameStringName of the space. Immutable after creation.
    tagStringIdentifies the instance of the space, changed each time it is modified. Generated by Dremio, immutable.
    createdAtStringRFC3339 date (example: 2017-10-27T21:08:22.858Z) representing the creation datetime. Immutable.
    children[CatalogEntitySummary]The children of the space. Optional, immutable.

    Folder

    Represents a folder in Dremio. All folders returned by the REST API will have an entityType of folder.

    The JSON representation of a folder looks like this:

    {
      "entityType": "folder" [immutable, generated by Dremio],
      "id": String [immutable, generated by Dremio],
      "path": [String] [immutable, generated by Dremio],
      "tag": String [immutable, generated by Dremio],
      "children": [CatalogEntitySummary] [optional] [immutable, generated by Dremio]
    }
    

    Fields

    NameTypeDescription
    idStringFolder ID. Generated by Dremio, immutable.
    path[String]Catalog Entity path. Immutable after creation.
    tagStringIdentifies the instance of the folder, changed each time it is modified. Generated by Dremio, immutable.
    createdAtStringRFC3339 date (example: 2017-10-27T21:08:22.858Z) representing the creation datetime. Immutable.
    children[CatalogEntitySummary]The children of the folder. Optional, immutable.

    File

    Represents a file in Dremio. All files returned by the REST API will have an entityType of file.

    The JSON representation of a file looks like this:

    {
      "entityType": "file" [immutable, generated by Dremio],
      "id": String [immutable, generated by Dremio],
      "path": [String] [immutable, generated by Dremio]
    }
    

    Fields

    NameTypeDescription
    idStringFile ID. Generated by Dremio, immutable.
    path[String]Catalog Entity path. Immutable after creation.

    Home

    Represents a home space in Dremio. All home spaces returned by the REST API will have an entityType of home.

    The JSON representation of a home space looks like this:

    {
      "entityType": "home" [immutable, generated by Dremio],
      "id": String [immutable, generated by Dremio],
      "name": String [immutable, generated by Dremio],
      "tag": String [immutable, generated by Dremio],
      "children": [CatalogEntitySummary] [optional] [immutable, generated by Dremio]
    }
    

    Fields

    NameTypeDescription
    idStringHome space ID. Generated by Dremio, immutable.
    nameStringName of the home space. Generated by Dremio, immutable.
    tagStringIdentifies the instance of the home space, changed each time it is modified. Generated by Dremio, immutable.
    children[CatalogEntitySummary]The children of the home space. Optional, immutable.

    Source

    Represents a source in Dremio. All sources returned by the REST API will have an entityType of source.

    The JSON representation of a source looks like this:

    {
      "entityType": "source" [immutable, generated by Dremio],
      "id": String [immutable, generated by Dremio],
      "name": String [immutable after creation],
      "description": String,
      "tag": String [immutable, generated by Dremio],
      "type": String (Source Type) [immutable after creation],
      "config": Object (Source Config),
      "createdAt": String (RFC3339 date) [immutable],
      "metadataPolicy": {
        "authTTLMs": Number,
        "datasetRefreshAfterMs": Number,
        "datasetExpireAfterMs": Number,
        "namesRefreshMs": Number,
        "datasetUpdateMode": String ["PREFETCH", "PREFETCH_QUERIED", "INLINE"]
      },
      "state": {
        "status": String ["good", "bad", "warn"] [immutable, generated by Dremio],
        "messages": [Status Message] [immutable, generated by Dremio]
      },
      "accelerationRefreshPeriodMs": Number,
      "accelerationGracePeriodMs": Number
    }
    

    Fields

    NameTypeDescription
    idStringSource ID. Generated by Dremio, immutable.
    nameStringName of the source. Immutable after creation.
    descriptionStringDetailed description.
    tagStringIdentifies the instance of the source, changed each time a source is modified. Generated by Dremio, immutable.
    typeStringSource type. Immutable after creation.
    configObjectSource config based on the specified source type.
    createdAtStringRFC3339 date (example: 2017-10-27T21:08:22.858Z) representing the creation datetime. Immutable.
    metadataPolicyMetadata PolicyPolicies regarding updating of a source’s metadata.
    stateSource StateState of the source. Generated by Dremio, immutable.
    accelerationRefreshPeriodMsNumberHow often to refresh reflections on data in the source.
    accelerationGracePeriodMsNumberHow long to keep reflections on data in the source before it expires.

    Metadata Policy

    Policies regarding updating of a source’s metadata.

    {
      "authTTLMs": Number,
      "datasetRefreshAfterMs": Number,
      "datasetExpireAfterMs": Number,
      "namesRefreshMs": Number,
      "datasetUpdateMode": String ["PREFETCH", "PREFETCH_QUERIED", "INLINE"]
    }
    

    Fields

    NameTypeDescription
    authTTLMsNumberAuthorization expiry in milliseconds.
    datasetRefreshAfterMsNumberHow often (in milliseconds) to refresh the metadata of datasets in the source.
    datasetExpireAfterMsNumberHow long (in miliseconds) to keep the metadata before it expires.
    namesRefreshMsNumberHow often (in milliseconds) to discover new datasets in the source.
    datasetUpdateModeStringMust be either PREFETCH, PREFETCH_QUERIED, or INLINE.

    PREFETCH - Dremio updates details for all datasets in a source. This mode increases query performance as less work needs to be done at query time.

    PREFETCH_QUERIED - Dremio updates details for previously queried objects in a source. This mode increases query performance as less work needs to be done at query time for these datasets.

    INLINE - Dremio updates details for a dataset at query time. This mode minimizes metadata queries on a source when not used, but might lead to longer planning times.

    Source State

    Represents the state of a source.

    {
      "status": String ["good", "bad", "warn"] [immutable],
      "messages": [StatusMessage] [immutable]
    }
    

    Fields

    NameTypeDescription
    statusStringEither info, bad, or warn. Immutable
    messagesStatusMessage[]An array of status messages. Immutable.

    Source Status Message

    Represents a status message for a source.

    {
      "level": String ["INFO", "WARN", "ERROR"] [immutable],
      "message": String [immutable]
    }
    

    Fields

    NameTypeDescription
    statusStringEither INFO, WARN, or ERROR. Immutable.
    messageStringStatus message. Immutable.

    Enterprise Edition Properties

    The Enterprise Edition of Dremio supports additional functionality that is exposed via the Catalog API.

    Access Control List

    Note:
    Introduced in Dremio 2.1.0

    Note:

    GET/PUT /accessControlList are deprecated for instances using Dremio v16.0+.

    Access control settings for the catalog entity. Exposed as a top level accessControlList property with the following data:

    {
      "users": [AccessControl] [optional],
      "groups": [AccessControl] [optional],
      "version": Number
    }
    

    Fields

    NameTypeDescription
    versionNumberIdentifies the instance, changed each time it is modified. Generated by Dremio, immutable.
    usersAccessControl[]List of user access controls.
    groupsAccessControl[]List of group access controls.

    Access Control

    {
      "id": String [READ, WRITE],
      "permissions": [String]
    }
    

    Fields

    NameTypeDescription
    idStringThe id of the user/group that the access control applies to.
    permissionsString[]List of permissions that the user/group has. Valid values are READ, WRITE.