On this page

    Source

    Deprecated:

    Use the Catalog API instead. The Source API will be removed in a future release.

    Represents a source in Dremio. All sources returned by the REST API will have a _type of source.

    The JSON representation of a source looks like this:

    {
    "_type": "source" [immutable, generated by Dremio],
    "id": String [immutable, generated by Dremio],
    "name": String [immutable after creation],
    "description": String,
    "tag": String [immutable, generated by Dremio],
    "type": String (Source Type) [immutable after creation],
    "config": Object (Source Config),
    "createdAt": String (RFC3339 date) [immutable],
    "metadataPolicy": {
    "authTTLMs": Number,
    "datasetRefreshAfterMs": Number,
    "datasetExpireAfterMs": Number,
    "namesRefreshMs": Number,
    "datasetUpdateMode": String ["PREFETCH", "PREFETCH_QUERIED", "INLINE"]
    },
    "state": {
    "status": String ["good", "bad", "warn"] [immutable, generated by Dremio],
    "messages": [Status Message] [immutable, generated by Dremio]
    },
    "accelerationRefreshPeriodMs": Number,
    "accelerationGracePeriodMs": Number
    }
    
    NameTypeDescription
    idStringSource ID. Generated by Dremio, immutable.
    nameStringName of the source. Immutable after creation.
    descriptionStringDetailed description. (optional)
    tagStringIdentifies the instance of the source, changed each time a source is modified. Generated by Dremio, immutable.
    typeStringSource type. Immutable after creation.
    configObjectSource config based on the specified source type.
    createdAtStringRFC3339 date (example: 2017-10-27T21:08:22.858Z) representing the creation datetime. Immutable.
    metadataPolicyMetadata PolicyPolicies regarding updating of a source’s metadata. (optional)
    stateSource StateState of the source. Generated by Dremio, immutable.
    accelerationRefreshPeriodMsNumberHow often to refresh reflections on data in the source. (optional)
    accelerationGracePeriodMsNumberHow long to keep reflections on data in the source before it expires. (optional)

    Metadata Policy

    Policies regarding updating of a source’s metadata.

    {
    "authTTLMs": Number,
    "datasetRefreshAfterMs": Number,
    "datasetExpireAfterMs": Number,
    "namesRefreshMs": Number,
    "datasetUpdateMode": String ["PREFETCH", "PREFETCH_QUERIED", "INLINE"]
    }
    
    NameTypeDescription
    authTTLMsNumberAuthorization expiry in milliseconds.
    datasetRefreshAfterMsNumberHow often (in milliseconds) to refresh the metadata of datasets in the source.
    datasetExpireAfterMsNumberHow long (in miliseconds) to keep the metadata before it expires.
    namesRefreshMsNumberHow often (in milliseconds) to discover new datasets in the source.
    datasetUpdateModeStringMust be either PREFETCH, PREFETCH_QUERIED, or INLINE.

    PREFETCH - Dremio updates details for all datasets in a source. This mode increases query performance as less work needs to be done at query time.

    PREFETCH_QUERIED - Dremio updates details for previously queried objects in a source. This mode increases query performance as less work needs to be done at query time for these datasets.

    INLINE - Dremio updates details for a dataset at query time. This mode minimizes metadata queries on a source when not used, but might lead to longer planning times.

    Source State

    Represents the state of a source.

    {
    "status": String ["good", "bad", "warn"] [immutable],
    "messages": [StatusMessage] [immutable]
    }
    
    NameTypeDescription
    statusStringEither good, bad, or warn. Immutable
    messagesStatusMessage[]An array of source status messages. Immutable.

    Source Status Message

    Represents a status message for a source.

    {
    "level": String ["INFO", "WARN", "ERROR"] [immutable],
    "message": String [immutable]
    }
    
    NameTypeDescription
    statusStringEither INFO, WARN, or ERROR. Immutable.
    messageStringStatus message. Immutable.