Skip to main content

Source

The Source API enables you to manage the sources that are supported in Dremio. You can add, retrieve, update, and delete sources. Sources are of entityType: source.

The primary Arctic catalog is created when you add a Sonar project and provides data management capabilities for your project. The primary Arctic catalog is considered a source entity within the API.

note

The source object contains the same attributes for all source types except for the config object, which is different for each source type. The examples on this page use an Amazon S3 source to demonstrate the available endpoints. For configuration information for each source, see Source Configuration.

Source Object for Amazon S3
{
"entityType": "source",
"config": {
"accessKey": "AKIAIOSFODNN7EXAMPLE",
"accessSecret": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
"secure": true,
"whitelistedBuckets": [
"sales",
"marketing"
],
"rootPath": "/",
"enableAsync": true,
"compatibilityMode": false,
"isCachingEnabled": true,
"maxCacheSpacePct": 100,
"requesterPays": false,
"enableFileStatusCheck": true,
"defaultCtasFormat": "ICEBERG",
"isPartitionInferenceEnabled": false,
"credentialType": "ACCESS_KEY"
},
"id": "4abb1367-5282-43e8-a0f6-54a8ae6428ed",
"tag": "676e73e0-8b51-44d0-972d-ef80c29ac0c6",
"type": "S3",
"name": "source-name",
"createdAt": "2022-07-26T19:56:28.826Z",
"metadataPolicy": {
"authTTLMs": 86400000,
"namesRefreshMs": 3600000,
"datasetRefreshAfterMs": 3600000,
"datasetExpireAfterMs": 10800000,
"datasetUpdateMode": "PREFETCH_QUERIED",
"deleteUnavailableDatasets": true,
"autoPromoteDatasets": false
},
"accelerationGracePeriodMs": 10800000,
"accelerationRefreshPeriodMs": 3600000,
"accelerationNeverExpire": false,
"accelerationNeverRefresh": false,
"children": [
{
"id": "dremio:/customer-records/sales",
"path": [
"customer-records",
"sales"
],
"type": "CONTAINER",
"containerType": "FOLDER"
},
{
"id": "dremio:/customer-records/marketing",
"path": [
"customer-records",
"marketing"
],
"type": "CONTAINER",
"containerType": "FOLDER"
},
],
"allowCrossSourceSelection": false,
"disableMetadataValidityCheck": false,
"accessControlList": {
"roles": [
{
"id": "1e5f38e4-8209-46dc-96f0-cfbd3276dbd8",
"permissions": [
"ALTER",
"CREATE_TABLE",
"DROP",
"INSERT",
"DELETE",
"UPDATE",
"TRUNCATE",
"VIEW_REFLECTION",
"ALTER_REFLECTION",
"MODIFY",
"MANAGE_GRANTS",
"SELECT"
]
}
],
"users": [
{
"id": "4deaf0e4-f05c-441c-bb37-d4d6e43fdec7",
"permissions": [
"VIEW_REFLECTION",
"SELECT"
]
}
]
},
"permissions": [
"READ",
"WRITE",
"ALTER_REFLECTION",
"SELECT",
"ALTER",
"VIEW_REFLECTION",
"MODIFY",
"MANAGE_GRANTS",
"CREATE_TABLE",
"DROP",
"EXTERNAL_QUERY",
"INSERT",
"TRUNCATE",
"DELETE",
"UPDATE",
"EXECUTE",
"CREATE_SOURCE",
"ALL"
],
"owner": {
"ownerId": "5c9c394d-7395-4687-97dd-beac4384c359",
"ownerType": "USER"
}
}

Source Attributes

entityType

String

Specifies the type of container. For sources, the type is source.


config

Object

Configuration options for the specified data source.


id

String (UUID)

The unique identifier that is generated to identify a source.

Example 4abb1367-5282-43e8-a0f6-54a8ae6428ed


tag

String (UUID)

Identifies the instance of the source. Dremio changes this tag whenever a change is made to the source. Immutable by the user.

Example 676e73e0-8b51-44d0-972d-ef80c29ac0c6


type

String

Identifies the source type that you are configuring. The source type determines the config parameters that display.

For more information, read Configuration Attributes.

Enum ARCTIC, AWSGLUE, MSSQL, MYSQL, ORACLE, POSTGRES, REDSHIFT, S3, SNOWFLAKE

name

String

The user-defined name of the source.

Example customer-records


createdAt

String

The date and time that the source was created.

Example 2022-07-26T19:56:28.826Z


metadataPolicy

Object

The policies covering the update of a source’s metadata.


accelerationGracePeriodMs

Number

Identifies the length of time to keep reflections for all the datasets in a source before these reflections expire. The default setting is 10800000 milliseconds or three hours. The minimum amount of time that is supported is 3600000 milliseconds or one hour.

For more information, read Setting the Expiration Policy for Reflections.

Example 10800000


accelerationRefreshPeriodMs

Number

Identifies the refresh frequency for a dataset's reflections. The default setting is 3600000 milliseconds or one hour, which is also the minimum amount of time that is supported.

Example 3600000


accelerationNeverExpire

Boolean

Option to set an expiration for accelerations. Default setting is false. Set to true to prevent accelerations from expiring and to override the accelerationGracePeriodMs setting.


accelerationNeverRefresh

Boolean

Option to set a refresh for accelerations. Default setting is false. Set to true to prevent accelerations from refreshing and to override the accelerationRefreshPeriodMs setting.


children

Object

The catalog entities (including datasets, files, and folders) contained in the source.


allowCrossSourceSelection

Boolean

For queries that can select from multiple sources, this option enables a source to be available in that query. Default setting is false. Set to true to enable this option.


disableMetadataValidityCheck

Boolean

Option to set a validity check for the source's metadata. Default setting is false. Set to true to disable the validity check.

Note: This attribute is not supported by default. Contact Dremio Support to enable the option.


accessControlList

Object

Information about users and roles with privileges on the source and the specific privileges each user or role has. May include a users array, a roles array, or both, depending on the configured access and privileges. The accessControlList object is empty if source-specific access control privileges are not set.

Note: For Arctic sources and the primary Arctic catalog, the source object does not include the accessControlList object.


permissions

Array

The permissions that the user submitting the API call has to a source. This will be an empty array, unless the query parameter include=permissions is set. When set, the permission values that are available will be based on the source you are connecting to.

For more information, read Privileges.


owner

Object

The owner of the source.

config

accessKey

String

A unique identifier from your AWS account used in conjunction with a secret access key to sign API requests made to your data source.

Example AKIAIOSFODNN7EXAMPLE


accessSecret

String

A secret access key from your AWS account that is used in conjunction with an access key to cryptographically sign programmatic requests. Signing a request identifies the sender and prevents the request from being altered.

Example wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY


secure

Boolean

Option to enable SSL encryption for the source. Default setting is true. Set to false to disable SSL encryption.


whitelistedBuckets

Array

A list of allowed Amazon S3 buckets.

Example sales, marketing


rootPath

String

Identifies the root path of the S3 bucket.

Example /


enableAsync

Boolean

Option to enable asynchronous access, which allows cloud caching for the source to support simultaneous actions such as adding and editing. Default setting is true. Set to false to disable asynchronous access.


compatibilityMode

Boolean

Option to enable compatibility mode for an Amazon S3 source, which allows you to use S3-compatible storage. Default setting is false. Set to true to enable compatibility mode.


isCachingEnabled

Boolean

Option to enable local caching. Default setting is true. Set to false to disable local caching.


maxCacheSpacePct

Number

Sets the maximum percent of total available cache space to use, when this space is available. The minimum value is 1 and the maximum value is 100. The default value is 100.

Example 100


requesterPays

Boolean

Option to apply S3 requests to the requestor. Default setting is false. Set to true to enable this option.


enableFileStatusCheck

Boolean

Option to enable file status check. Default setting is true. Set to false to disable file status check.


defaultCtasFormat

String

Sets the default format for tables that are created in Dremio. The default format that is used is ICEBERG.

Enum ICEBERG, PARQUET

isPartitionInferenceEnabled

Boolean

Option to enable partition column inference. The default setting is false. Set to true to enable partition column inference.


credentialType

String

Specifies the type of credential to enable Dremio to connect to the source.

Enum ACCESS_KEY, PROJECT_DATA_CRED

metadataPolicy

authTTLMs

Number

Sets the length of time, in milliseconds, that source permissions are cached. For example, if the default 24 hours (86400000 milliseconds) is used, then for each file you query, your permission status is checked once every 24 hours. The minimum supported time period is one minute (60000 milliseconds).

Example 86400000


namesRefreshMs

Number

Sets when to run a refresh of a source, in milliseconds. The default time period is one hour (3600000 milliseconds). The minimum supported time period is one minute (60000 milliseconds).

Example 3600000


datasetRefreshAfterMs

Number

Determines how often the metadata in the dataset is refreshed, in milliseconds. The default refresh rate is one hour (3600000 milliseconds). The minimum supported refresh rate is one minute (60000 milliseconds).

Example 3600000


datasetExpireAfterMs

Number

Sets the amount of time, in milliseconds, to keep the metadata before it expires. The default time period is one hour (3600000 milliseconds). The minimum supported time period is one minute (60000 milliseconds).

Example 10800000


datasetUpdateMode

String

Sets the metadata policy for when a dataset is updated. Use PREFETCH_QUERIED to update the details for previously queried objects in a source.

Enum PREFETCH_QUERIED

deleteUnavailableDatasets

Boolean

Option to remove dataset definitions if the underlying data is unavailable to Dremio. The default setting is true. Set to false to keep the dataset definitions.


autoPromoteDatasets

Boolean

Option to automatically format files into tables when a query is issued. The default setting is false. Set to true to enable this capability.

Note: This attribute applies only to metastore and object storage sources.

children

id

String

The catalog entity ID, which is generated by Dremio and is immutable.

Example dremio:/customer-records/sales


path

String

The catalog entity path, which is immutable.

Example customer-records


type

String

The type of the object that is contained in the source. This typing is generated by Dremio and is immutable. For folders, the type is CONTAINER. For datasets, the type is DATASET. For Parquet and other file formats, the type is FILE.

Enum CONTAINER, DATASET, FILE

containerType

String

Identifies the container type. This attribute is displayed only if type is set to CONTAINER. This attribute is automatically generated by Dremio and is immutable.

Enum FOLDER

datasetType

String

Identifies the dataset type. This attribute is displayed only if type is set to DATASET. The dataset can be one of the following types:
DIRECT: A dataset that is from an external source (such as PostgreSQL).
PROMOTED: A table that is promoted from a source.
VIRTUAL: A view that is created on another dataset.

This attribute is automatically generated by Dremio and is immutable.

Enum DIRECT, PROMOTED, VIRTUAL

accessControlList

roles

Array

Information about the roles that have been granted privileges on the source and the privileges each role has.


users

Array

Information about the users that have been granted privileges on the source and the privileges each user has.

roles

id

String

The ID of the role.

Example 1e5f38e4-8209-46dc-96f0-cfbd3276dbd8


permissions

Array

The privileges that the role has on the source.

For more information, read Privileges.

users

id

String

The ID of the user.

Example 4deaf0e4-f05c-441c-bb37-d4d6e43fdec7


permissions

Array

The privileges that the user has on the source.

For more information, read Privileges.

owner

ownerId

String (UUID)

The unique identifier for the owner (user or role) of the source. This identifier corresponds to the user_id or role_id in the users or roles system table.

For more information, read the Users system table.

Example 5c9c394d-7395-4687-97dd-beac4384c359


ownerType

String

The type of the owner.

Enum USER, ROLE

Adding a Source

Add a source.

Adding a Source
POST  /v0/projects/{project-id}/catalog

Parameters

project-id

path

String (UUID)

The unique identifier for the project that contains the source that you want to add.

Example 02d36975-73eb-47ed-9bb5-de73060380f6


entityType

body

String

Specifies the type of container. For sources, the type is source.


config

body

Object

Configuration options for the specified data source.


type

body

String

Identifies the source type that you are configuring. The source type determines the config parameters that display.

For more information, read Configuration Attributes.

Enum ARCTIC, AWSGLUE, MSSQL, MYSQL, ORACLE, POSTGRES, REDSHIFT, S3, SNOWFLAKE

name

body

String

The user-defined name of the source.

Example customer-records


metadataPolicy

body

Source Metadata

The policies covering the update of a source’s metadata.


accelerationGracePeriodMs

body

Number

Identifies the length of time to keep reflections for all the datasets in a source before these reflections expire. The default setting is 10800000 milliseconds or three hours. The minimum amount of time that is supported is 3600000 milliseconds or one hour.

For more information, read Setting the Expiration Policy for Reflections.

Example 10800000


accelerationRefreshPeriodMs

body

Number

Identifies the refresh frequency for a dataset's reflections. The default setting is 3600000 milliseconds or one hour, which is also the minimum amount of time that is supported.

Example 3600000


accessControlList

body

Object

Object used to specify which users and roles should have privileges on the source and the specific privileges each user and role has. May include a users array, a roles array, or both. Not supported for Arctic sources and the primary Arctic catalog.

config

hostname

body

String

The name of the server you are connecting to.

Example 177.15.0.112


port

body

String

The port number of the server you are connecting to.

Example 3306


authenticationType

body

String

The type of authentication needed to connect to the server. Use ANONYMOUS when no authentication is needed and MASTER when credentials from a master database user is required.

Enum ANONYMOUS, MASTER

netWriteTimeout

body

Number

The amount of time, in seconds, to wait for data from the server before aborting the connection.

Example 60


fetchSize

body

Number

The number of records to fetch per request. Set to 0 to enable Dremio to automatically set the size.

Example 200


maxIdleConns

body

Number

Sets the maximum number of idle connections that you want to keep.

Example 8


idleTimeSec

body

Number

Sets the idle time, in seconds, before a connection is evaluated for closure.

Example 60


propertyList

body

Array

An array of name/value pairs defining optional connection properties that is used by the source. Use a comma separated list to specify multiple name value pairs.

metadataPolicy

authTTLMs

body

Number

Sets the expiration for an authorization to a metadata policy, in milliseconds. The default time period is 24 hours (86400000 milliseconds). The minimum supported time period is one minute (60000 milliseconds).

Example 86400000


namesRefreshMs

body

Number

Sets when to run a refresh of a source, in milliseconds. The default time period is one hour (3600000 milliseconds). The minimum supported time period is one minute (60000 milliseconds).

Example 86400000


datasetRefreshAfterMs

body

Number

Determines how often the metadata in the dataset is refreshed, in milliseconds. The default refresh rate is one hour (3600000 milliseconds). The minimum supported refresh rate is one minute (60000 milliseconds).

Example 86400000


datasetExpireAfterMs

body

Number

Sets the amount of time, in milliseconds, to keep the metadata before it expires. The default time period is one hour (3600000 milliseconds). The minimum supported time period is one minute (60000 milliseconds).

Example 259200000


datasetUpdateMode

body

String

Sets the metadata policy for when a dataset is updated. Use PREFETCH_QUERIED to update the details for previously queried objects in a source.

Enum PREFETCH_QUERIED

deleteUnavailableDatasets

body

Boolean

Option to remove dataset definitions if the underlying data is unavailable to Dremio, for example, at the time the data is queried. The default setting is true. Set to false to keep the dataset definitions.

accessControlList

roles

body

Array

The roles that should have privileges on the source and the privileges each role should have.


users

body

Array

The users that should have privileges on the source and the privileges each user should have.

roles

id

body

String

The ID of the role.

Example 9ab42853-bdef-465f-b9bb-381a13a9bf78


permissions

body

Array

The privileges that the role should have on the source.

For more information, read Privileges.

users

id

body

String

The ID of the user.


permissions

body

Array

The privileges that the user should have on the source.

For more information, read Privileges.

Example Request
curl -X POST 'https://api.dremio.cloud/v0/projects/02d36975-73eb-47ed-9bb5-de73060380f6/catalog' \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json' \
-d '{
"entityType": "source",
"config": {
"hostname": "177.15.0.112",
"port": "3306",
"authenticationType": "ANONYMOUS",
"netWriteTimeout": 60,
"fetchSize": 200,
"maxIdleConns": 8,
"idleTimeSec": 60,
"propertyList": []
},
"type": "MYSQL",
"name": "recruitingdb",
"metadataPolicy": {
"authTTLMs": 86400000,
"namesRefreshMs": 86400000,
"datasetRefreshAfterMs": 86400000,
"datasetExpireAfterMs": 259200000,
"datasetUpdateMode": "PREFETCH_QUERIED",
"deleteUnavailableDatasets": true
},
"accelerationGracePeriodMs": 10800000,
"accelerationRefreshPeriodMs": 3600000,
"accessControlList": {
"roles": [
{
"id": "9ab42853-bdef-465f-b9bb-381a13a9bf78",
"permissions": []
}
],
"users": []
}
}'
Example Response
{
"entityType": "source",
"config": {
"hostname": "177.15.0.112",
"port": "3306",
"authenticationType": "ANONYMOUS",
"netWriteTimeout": 60,
"fetchSize": 200,
"maxIdleConns": 8,
"idleTimeSec": 60,
"propertyList": []
},
"id": "629904cf-9c06-4ae6-8cc1-faf1d2f1ab5f",
"tag": "a927dd14-2c7d-4684-9de2-4000e05c0d07",
"type": "MYSQL",
"name": "recruitingdb",
"createdAt": "2022-09-12T07:22:19.334Z",
"metadataPolicy": {
"authTTLMs": 86400000,
"namesRefreshMs": 86400000,
"datasetRefreshAfterMs": 86400000,
"datasetExpireAfterMs": 259200000,
"datasetUpdateMode": "PREFETCH_QUERIED",
"deleteUnavailableDatasets": true
},
"accelerationGracePeriodMs": 10800000,
"accelerationRefreshPeriodMs": 3600000,
"accelerationNeverExpire": false,
"accelerationNeverRefresh": false,
"children": [
{
"id": "42312a90-1723-4bcb-b62c-bf9abee371b1",
"path": [
"recruitingdb",
"contacts"
],
"type": "CONTAINER",
"containerType": "FOLDER"
}
],
"allowCrossSourceSelection": false,
"disableMetadataValidityCheck": false,
"accessControlList": {
"roles": [
{
"id": "9ab42853-bdef-465f-b9bb-381a13a9bf78",
"permissions": []
}
],
"users": []
},
"permissions": [],
"checkTableAuthorizer": false,
"owner": {
"ownerId": "5c9c394d-7395-4687-97dd-beac4384c359",
"ownerType": "USER"
}
}

Responses

200

Source created successfully

400

Bad input parameter

401

Unauthorized access


Retrieving a Source by ID

Get information about the source using the source ID.

Retrieving a Source by ID
GET /v0/projects/{project-id}/catalog/{id}

Parameters

project-id

path

String (UUID)

The unique identifier for the project that contains the source.

Example 02d36975-73eb-47ed-9bb5-de73060380f6


id

path

String (UUID)

The unique identifier for the source that you want to retrieve.

Example ffbe8c1d-1db7-48d1-9c58-f452838fedc0


include

query

String

Optional

When using include, the name of a non-default response field is returned. You can retrieve either a list of access control list names - aclNames - or a list of permissions. Read Include and Exclude Query Parameters for usage examples.

Enum aclNames, permissions

Example ffbe8c1d-1db7-48d1-9c58-f452838fedc0?include=permissions


exclude

query

String

Optional

When using exclude, the children field is excluded from the response. Read Include and Exclude Query Parameters for usage examples.

Enum children

Example ffbe8c1d-1db7-48d1-9c58-f452838fedc0?exclude=children

Example Request
curl -X GET 'https://api.dremio.cloud/v0/projects/02d36975-73eb-47ed-9bb5-de73060380f6/catalog/ffbe8c1d-1db7-48d1-9c58-f452838fedc0' \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json'
Example Response
{
"entityType": "source",
"config": {
"hostname": "177.15.0.112",
"port": "3306",
"authenticationType": "ANONYMOUS",
"netWriteTimeout": 60,
"fetchSize": 200,
"maxIdleConns": 8,
"idleTimeSec": 60,
"propertyList": []
},
"id": "629904cf-9c06-4ae6-8cc1-faf1d2f1ab5f",
"tag": "a927dd14-2c7d-4684-9de2-4000e05c0d07",
"type": "MYSQL",
"name": "recruitingdb",
"createdAt": "2022-09-12T07:22:19.334Z",
"metadataPolicy": {
"authTTLMs": 86400000,
"namesRefreshMs": 86400000,
"datasetRefreshAfterMs": 86400000,
"datasetExpireAfterMs": 259200000,
"datasetUpdateMode": "PREFETCH_QUERIED",
"deleteUnavailableDatasets": true
},
"accelerationGracePeriodMs": 10800000,
"accelerationRefreshPeriodMs": 3600000,
"accelerationNeverExpire": false,
"accelerationNeverRefresh": false,
"children": [
{
"id": "42312a90-1723-4bcb-b62c-bf9abee371b1",
"path": [
"recruitingdb",
"contacts"
],
"type": "CONTAINER",
"containerType": "FOLDER"
}
],
"allowCrossSourceSelection": false,
"disableMetadataValidityCheck": false,
"accessControlList": {},
"permissions": [],
"checkTableAuthorizer": false,
"owner": {
"ownerId": "5c9c394d-7395-4687-97dd-beac4384c359",
"ownerType": "USER"
}
}

Responses

200

Source returned successfully

400

Bad input parameter

401

Unauthorized access

404

Not found


Retrieving a Source by Path

Get information about the source using the source's path.

Retrieving a Source by Path
GET /v0/projects/{project-id}/catalog/by-path/{path}

Parameters

project-id

path

String (UUID)

The unique identifier for the project that contains the source.

Example 02d36975-73eb-47ed-9bb5-de73060380f6


path

path

String (UUID)

Source's location within Dremio, using forward slashes as separators. For example, for the source "Samples," the path is Samples. If the name of any component in the path includes special characters for URLs, such as spaces, use URL encoding to replace the special characters with their UTF-8-equivalent characters. For example, "Dremio University" should be Dremio%20University in the URL path.

Example recruitingdb

Example Request
curl -X GET 'https://api.dremio.cloud/v0/projects/02d36975-73eb-47ed-9bb5-de73060380f6/catalog/by-path/recruitingdb' \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json'
Example Response
{
"entityType": "source",
"config": {
"hostname": "177.15.0.112",
"port": "3306",
"authenticationType": "ANONYMOUS",
"netWriteTimeout": 60,
"fetchSize": 200,
"maxIdleConns": 8,
"idleTimeSec": 60,
"propertyList": []
},
"id": "629904cf-9c06-4ae6-8cc1-faf1d2f1ab5f",
"tag": "a927dd14-2c7d-4684-9de2-4000e05c0d07",
"type": "MYSQL",
"name": "recruitingdb",
"createdAt": "2022-09-12T07:22:19.334Z",
"metadataPolicy": {
"authTTLMs": 86400000,
"namesRefreshMs": 86400000,
"datasetRefreshAfterMs": 86400000,
"datasetExpireAfterMs": 259200000,
"datasetUpdateMode": "PREFETCH_QUERIED",
"deleteUnavailableDatasets": true
},
"accelerationGracePeriodMs": 10800000,
"accelerationRefreshPeriodMs": 3600000,
"accelerationNeverExpire": false,
"accelerationNeverRefresh": false,
"children": [
{
"id": "42312a90-1723-4bcb-b62c-bf9abee371b1",
"path": [
"recruitingdb",
"contacts"
],
"type": "CONTAINER",
"containerType": "FOLDER"
}
],
"allowCrossSourceSelection": false,
"disableMetadataValidityCheck": false,
"accessControlList": {},
"permissions": [],
"checkTableAuthorizer": false,
"owner": {
"ownerId": "5c9c394d-7395-4687-97dd-beac4384c359",
"ownerType": "USER"
}
}

Responses

200

Source returned successfully

400

Bad input parameter

401

Unauthorized access

404

Not found


Updating a Source

Edit the source information using the source ID.

Updating a Source
PUT /v0/projects/{project-id}/catalog/{id}

Parameters

project-id

path

String (UUID)

The unique identifier for the project that contains the source.

Example 02d36975-73eb-47ed-9bb5-de73060380f6


id

path

String (UUID)

The unique identifier for the source that you want to update.

Example ffbe8c1d-1db7-48d1-9c58-f452838fedc0


entityType

body

String

Specifies the type of container. For sources, the type is source.


config

body

Object

Configuration options for the specified data source.


type

body

String

Identifies the source type that you are configuring. The source type determines the config parameters that display.

For more information, read Configuration Attributes.

Enum ARCTIC, AWSGLUE, MSSQL, MYSQL, ORACLE, POSTGRES, REDSHIFT, S3, SNOWFLAKE

name

body

String

The user-defined name of the source.

Example recruitingdb


metadataPolicy

body

Source Metadata

The policies covering the update of a source’s metadata.


accelerationGracePeriodMs

body

Number

Identifies the length of time to keep reflections for all the datasets in a source before these reflections expire. The default setting is 10800000 milliseconds or three hours. The minimum amount of time that is supported is 3600000 milliseconds or one hour.

For more information, read Setting the Expiration Policy for Reflections.

Example 10800000


accelerationRefreshPeriodMs

body

Number

Identifies the refresh frequency for a dataset's reflections. The default setting is 3600000 milliseconds or one hour, which is also the minimum amount of time that is supported.

Example 3600000


accessControlList

body

Object

Object used to specify which users and roles should have privileges on the source and the specific privileges each user and role has. May include a users array, a roles array, or both. Not supported for Arctic sources and the primary Arctic catalog.

config

hostname

body

String

The name of the server you are connecting to.

Example 177.15.0.112


port

body

String

The port number of the server you are connecting to.

Example 3306


authenticationType

body

String

The type of authentication needed to connect to the server. Use ANONYMOUS when no authentication is needed and MASTER when credentials from a master database user is required.

Enum ANONYMOUS, MASTER

netWriteTimeout

body

Number

The amount of time, in seconds, to wait for data from the server before aborting the connection.

Example 60


fetchSize

body

Number

The number of records to fetch per request. Set to 0 to enable Dremio to automatically set the size.

Example 200


maxIdleConns

body

Number

Sets the maximum number of idle connections that you want to keep.

Example 8


idleTimeSec

body

Number

Sets the idle time, in seconds, before a connection is evaluated for closure.

Example 60


propertyList

body

Array

An array of name/value pairs defining optional connection properties that is used by the source. Use a comma separated list to specify multiple name value pairs.

metadataPolicy

authTTLMs

body

Number

Sets the expiration for an authorization to a metadata policy, in milliseconds. The default time period is 24 hours (86400000 milliseconds). The minimum supported time period is one minute (60000 milliseconds).

Example 86400000


namesRefreshMs

body

Number

Sets when to run a refresh of a source, in milliseconds. The default time period is one hour (3600000 milliseconds). The minimum supported time period is one minute (60000 milliseconds).

Example 86400000


datasetRefreshAfterMs

body

Number

Determines how often the metadata in the dataset is refreshed, in milliseconds. The default refresh rate is one hour (3600000 milliseconds). The minimum supported refresh rate is one minute (60000 milliseconds).

Example 86400000


datasetExpireAfterMs

body

Number

Sets the amount of time, in milliseconds, to keep the metadata before it expires. The default time period is one hour (3600000 milliseconds). The minimum supported time period is one minute (60000 milliseconds).

Example 259200000


datasetUpdateMode

body

String

Sets the metadata policy for when a dataset is updated. Use PREFETCH_QUERIED to update the details for previously queried objects in a source.

Enum PREFETCH_QUERIED

deleteUnavailableDatasets

body

Boolean

Option to remove dataset definitions if the underlying data is unavailable to Dremio, for example, at the time the data is queried. The default setting is true. Set to false to keep the dataset definitions.

accessControlList

roles

body

Array

The roles that should have privileges on the source and the privileges each role should have.


users

body

Array

The users that should have privileges on the source and the privileges each user should have.

roles

id

body

String

The ID of the role.

Example 9ab42853-bdef-465f-b9bb-381a13a9bf78


permissions

body

Array

The privileges that the role should have on the source.

For more information, read Privileges.

users

id

body

String

The ID of the user.


permissions

body

Array

The privileges that the user should have on the source.

For more information, read Privileges.

Example Request
curl -X GET 'https://api.dremio.cloud/v0/projects/02d36975-73eb-47ed-9bb5-de73060380f6/catalog/ffbe8c1d-1db7-48d1-9c58-f452838fedc0' \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json'
-d '{
"entityType": "source",
"config": {
"hostname": "177.15.0.112",
"port": "3306",
"authenticationType": "ANONYMOUS",
"netWriteTimeout": 60,
"fetchSize": 200,
"maxIdleConns": 8,
"idleTimeSec": 60,
"propertyList": []
},
"type": "MYSQL",
"name": "recruitingdb",
"metadataPolicy": {
"authTTLMs": 86400000,
"namesRefreshMs": 86400000,
"datasetRefreshAfterMs": 86400000,
"datasetExpireAfterMs": 259200000,
"datasetUpdateMode": "PREFETCH_QUERIED",
"deleteUnavailableDatasets": true
},
"accelerationGracePeriodMs": 10800000,
"accelerationRefreshPeriodMs": 3600000,
"accessControlList": {
"roles": [
{
"id": "9ab42853-bdef-465f-b9bb-381a13a9bf78",
"permissions": []
}
],
"users": []
}
}'
Example Response
{
"entityType": "source",
"config": {
"hostname": "177.15.0.112",
"port": "3306",
"authenticationType": "ANONYMOUS",
"netWriteTimeout": 60,
"fetchSize": 200,
"maxIdleConns": 8,
"idleTimeSec": 60,
"propertyList": []
},
"id": "629904cf-9c06-4ae6-8cc1-faf1d2f1ab5f",
"tag": "a927dd14-2c7d-4684-9de2-4000e05c0d07",
"type": "MYSQL",
"name": "recruitingdb",
"createdAt": "2022-09-12T07:22:19.334Z",
"metadataPolicy": {
"authTTLMs": 86400000,
"namesRefreshMs": 86400000,
"datasetRefreshAfterMs": 86400000,
"datasetExpireAfterMs": 259200000,
"datasetUpdateMode": "PREFETCH_QUERIED",
"deleteUnavailableDatasets": true
},
"accelerationGracePeriodMs": 10800000,
"accelerationRefreshPeriodMs": 3600000,
"accelerationNeverExpire": false,
"accelerationNeverRefresh": false,
"children": [
{
"id": "42312a90-1723-4bcb-b62c-bf9abee371b1",
"path": [
"recruitingdb",
"contacts"
],
"type": "CONTAINER",
"containerType": "FOLDER"
}
],
"allowCrossSourceSelection": false,
"disableMetadataValidityCheck": false,
"accessControlList": {
"roles": [
{
"id": "9ab42853-bdef-465f-b9bb-381a13a9bf78",
"permissions": []
}
],
"users": []
},
"permissions": [],
"checkTableAuthorizer": false,
"owner": {
"ownerId": "5c9c394d-7395-4687-97dd-beac4384c359",
"ownerType": "USER"
}
}

Responses

204

No content

400

Bad input parameter

401

Unauthorized access

404

Not found


Deleting a Source

Delete a source.

note

Dremio does not support deleting the primary Arctic catalog.

Deleting a Source
DELETE /v0/projects/{project-id}/catalog/{id}

Parameters

project-id

path

String (UUID)

The unique identifier for the project that contains the source.

Example 02d36975-73eb-47ed-9bb5-de73060380f6


id

path

String (UUID)

The unique identifier for the source that you want to delete. Deletion is not supported for the primary Arctic catalog.

Example 21a74cb1-3fcf-48c6-8f69-94c257ec3da3

Example Request
curl -X DELETE 'https://api.dremio.cloud/v0/projects/02d36975-73eb-47ed-9bb5-de73060380f6/catalog/21a74cb1-3fcf-48c6-8f69-94c257ec3da3' \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json'
Example Response
No response

Responses

204

No content

400

Bad input parameter

401

Unauthorized access

404

Not found