Source
The Source API enables you to manage the sources that are supported in Dremio. You can add, retrieve, update, and delete sources. Sources are of entityType: source
.
The primary Arctic catalog is created when you add a Sonar project and provides data management capabilities for your project. The primary Arctic catalog is considered a source entity within the API.
The source object contains the same attributes for all source types except for the config object, which is different for each source type. The examples on this page use an Amazon S3 source to demonstrate the available endpoints. For configuration information for each source, see Source Configuration.
{
"entityType": "source",
"config": {
"accessKey": "AKIAIOSFODNN7EXAMPLE",
"accessSecret": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
"secure": true,
"whitelistedBuckets": [
"sales",
"marketing"
],
"rootPath": "/",
"enableAsync": true,
"compatibilityMode": false,
"isCachingEnabled": true,
"maxCacheSpacePct": 100,
"requesterPays": false,
"enableFileStatusCheck": true,
"defaultCtasFormat": "ICEBERG",
"isPartitionInferenceEnabled": false,
"credentialType": "ACCESS_KEY"
},
"id": "4abb1367-5282-43e8-a0f6-54a8ae6428ed",
"tag": "676e73e0-8b51-44d0-972d-ef80c29ac0c6",
"type": "S3",
"name": "source-name",
"createdAt": "2022-07-26T19:56:28.826Z",
"metadataPolicy": {
"authTTLMs": 86400000,
"namesRefreshMs": 3600000,
"datasetRefreshAfterMs": 3600000,
"datasetExpireAfterMs": 10800000,
"datasetUpdateMode": "PREFETCH_QUERIED",
"deleteUnavailableDatasets": true,
"autoPromoteDatasets": false
},
"accelerationGracePeriodMs": 10800000,
"accelerationRefreshPeriodMs": 3600000,
"accelerationNeverExpire": false,
"accelerationNeverRefresh": false,
"children": [
{
"id": "dremio:/customer-records/sales",
"path": [
"customer-records",
"sales"
],
"type": "CONTAINER",
"containerType": "FOLDER"
},
{
"id": "dremio:/customer-records/marketing",
"path": [
"customer-records",
"marketing"
],
"type": "CONTAINER",
"containerType": "FOLDER"
},
],
"allowCrossSourceSelection": false,
"disableMetadataValidityCheck": false,
"accessControlList": {
"roles": [
{
"id": "1e5f38e4-8209-46dc-96f0-cfbd3276dbd8",
"permissions": [
"ALTER",
"CREATE_TABLE",
"DROP",
"INSERT",
"DELETE",
"UPDATE",
"TRUNCATE",
"VIEW_REFLECTION",
"ALTER_REFLECTION",
"MODIFY",
"MANAGE_GRANTS",
"SELECT"
]
}
],
"users": [
{
"id": "4deaf0e4-f05c-441c-bb37-d4d6e43fdec7",
"permissions": [
"VIEW_REFLECTION",
"SELECT"
]
}
]
},
"permissions": [
"READ",
"WRITE",
"ALTER_REFLECTION",
"SELECT",
"ALTER",
"VIEW_REFLECTION",
"MODIFY",
"MANAGE_GRANTS",
"CREATE_TABLE",
"DROP",
"EXTERNAL_QUERY",
"INSERT",
"TRUNCATE",
"DELETE",
"UPDATE",
"EXECUTE",
"CREATE_SOURCE",
"ALL"
],
"owner": {
"ownerId": "5c9c394d-7395-4687-97dd-beac4384c359",
"ownerType": "USER"
}
}
Source Attributes
entityType
String
Specifies the type of container. For sources, the type is source
.
Object
Configuration options for the specified data source.
id
String (UUID)
The unique identifier that is generated to identify a source.
Example 4abb1367-5282-43e8-a0f6-54a8ae6428ed
tag
String (UUID)
Identifies the instance of the source. Dremio changes this tag whenever a change is made to the source. Immutable by the user.
Example 676e73e0-8b51-44d0-972d-ef80c29ac0c6
type
String
Identifies the source type that you are configuring. The source type determines the config
parameters that display.
For more information, read Configuration Attributes.
EnumARCTIC, AWSGLUE, AZURE_STORAGE, MSSQL, MYSQL, ORACLE, POSTGRES, REDSHIFT, S3, SNOWFLAKE
name
String
The user-defined name of the source.
Example customer-records
createdAt
String
The date and time that the source was created.
Example 2022-07-26T19:56:28.826Z
Object
The policies covering the update of a source’s metadata.
accelerationGracePeriodMs
Number
Identifies the length of time to keep reflections for all the datasets in a source before these reflections expire. The default setting is 10800000
milliseconds or three hours. The minimum amount of time that is supported is 3600000
milliseconds or one hour.
For more information, read Setting the Expiration Policy for Reflections.
Example 10800000
accelerationRefreshPeriodMs
Number
Identifies the refresh frequency for a dataset's reflections. The default setting is 3600000
milliseconds or one hour, which is also the minimum amount of time that is supported.
Example 3600000
accelerationNeverExpire
Boolean
Option to set an expiration for accelerations. Default setting is false
. Set to true
to prevent accelerations from expiring and to override the accelerationGracePeriodMs
setting.
accelerationNeverRefresh
Boolean
Option to set a refresh for accelerations. Default setting is false
. Set to true
to prevent accelerations from refreshing and to override the accelerationRefreshPeriodMs
setting.
Object
The catalog entities (including datasets, files, and folders) contained in the source.
allowCrossSourceSelection
Boolean
For queries that can select from multiple sources, this option enables a source to be available in that query. Default setting is false
. Set to true
to enable this option.
disableMetadataValidityCheck
Boolean
Option to set a validity check for the source's metadata. Default setting is false
. Set to true
to disable the validity check.
Note: This attribute is not supported by default. Contact Dremio Support to enable the option.
Object
Information about users and roles with privileges on the source and the specific privileges each user or role has. May include a users array, a roles array, or both, depending on the configured access and privileges. The accessControlList object is empty if source-specific access control privileges are not set.
Note: For Arctic sources and the primary Arctic catalog, the source object does not include the accessControlList object.
permissions
Array
The permissions that the user submitting the API call has to a source. This will be an empty array, unless the query parameter include=permissions
is set. When set, the permission values that are available will be based on the source you are connecting to.
For more information, read Privileges.
Object
The owner of the source.
config
accessKey
String
A unique identifier from your AWS account used in conjunction with a secret access key to sign API requests made to your data source.
Example AKIAIOSFODNN7EXAMPLE
accessSecret
String
A secret access key from your AWS account that is used in conjunction with an access key to cryptographically sign programmatic requests. Signing a request identifies the sender and prevents the request from being altered.
Example wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
secure
Boolean
Option to enable SSL encryption for the source. Default setting is true
. Set to false
to disable SSL encryption.
whitelistedBuckets
Array
A list of allowed Amazon S3 buckets.
Example sales, marketing
rootPath
String
Identifies the root path of the S3 bucket.
Example /
enableAsync
Boolean
Option to enable asynchronous access, which allows cloud caching for the source to support simultaneous actions such as adding and editing. Default setting is true
. Set to false
to disable asynchronous access.
compatibilityMode
Boolean
Option to enable compatibility mode for an Amazon S3 source, which allows you to use S3-compatible storage. Default setting is false
. Set to true
to enable compatibility mode.
isCachingEnabled
Boolean
Option to enable local caching. Default setting is true
. Set to false
to disable local caching.
maxCacheSpacePct
Number
Sets the maximum percent of total available cache space to use, when this space is available. The minimum value is 1
and the maximum value is 100
. The default value is 100
.
Example 100
requesterPays
Boolean
Option to apply S3 requests to the requestor. Default setting is false
. Set to true
to enable this option.
enableFileStatusCheck
Boolean
Option to enable file status check. Default setting is true
. Set to false
to disable file status check.
defaultCtasFormat
String
Sets the default format for tables that are created in Dremio. The default format that is used is ICEBERG
.
ICEBERG, PARQUET
isPartitionInferenceEnabled
Boolean
Option to enable partition column inference. The default setting is false
. Set to true
to enable partition column inference.
credentialType
String
Specifies the type of credential to enable Dremio to connect to the source.
EnumACCESS_KEY, PROJECT_DATA_CRED
metadataPolicy
authTTLMs
Number
Sets the length of time, in milliseconds, that source permissions are cached. For example, if the default 24 hours (86400000
milliseconds) is used, then for each file you query, your permission status is checked once every 24 hours. The minimum supported time period is one minute (60000
milliseconds).
Example 86400000
namesRefreshMs
Number
Sets when to run a refresh of a source, in milliseconds. The default time period is one hour (3600000
milliseconds). The minimum supported time period is one minute (60000
milliseconds).
Example 3600000
datasetRefreshAfterMs
Number
Determines how often the metadata in the dataset is refreshed, in milliseconds. The default refresh rate is one hour (3600000
milliseconds). The minimum supported refresh rate is one minute (60000
milliseconds).
Example 3600000
datasetExpireAfterMs
Number
Sets the amount of time, in milliseconds, to keep the metadata before it expires. The default time period is one hour (3600000
milliseconds). The minimum supported time period is one minute (60000
milliseconds).
Example 10800000
datasetUpdateMode
String
Sets the metadata policy for when a dataset is updated. Use PREFETCH_QUERIED
to update the details for previously queried objects in a source.
PREFETCH_QUERIED
deleteUnavailableDatasets
Boolean
Option to remove dataset definitions if the underlying data is unavailable to Dremio. The default setting is true
. Set to false
to keep the dataset definitions.
autoPromoteDatasets
Boolean
Option to automatically format files into tables when a query is issued. The default setting is false
. Set to true
to enable this capability.
Note: This attribute applies only to metastore and object storage sources.
children
id
String
The catalog entity ID, which is generated by Dremio and is immutable.
Example dremio:/customer-records/sales
path
String
The catalog entity path, which is immutable.
Example customer-records
type
String
The type of the object that is contained in the source. This typing is generated by Dremio and is immutable. For folders, the type is CONTAINER
. For datasets, the type is DATASET
. For Parquet and other file formats, the type is FILE
.
CONTAINER, DATASET, FILE
containerType
String
Identifies the container type. This attribute is displayed only if type
is set to CONTAINER
. This attribute is automatically generated by Dremio and is immutable.
FOLDER
datasetType
String
Identifies the dataset type. This attribute is displayed only if type
is set to DATASET
. The dataset can be one of the following types:
• DIRECT
: A dataset that is from an external source (such as PostgreSQL).
• PROMOTED
: A table that is promoted from a source.
• VIRTUAL
: A view that is created on another dataset.
This attribute is automatically generated by Dremio and is immutable.
DIRECT, PROMOTED, VIRTUAL
accessControlList
Array
Information about the roles that have been granted privileges on the source and the privileges each role has.
Array
Information about the users that have been granted privileges on the source and the privileges each user has.
roles
id
String
The ID of the role.
Example 1e5f38e4-8209-46dc-96f0-cfbd3276dbd8
permissions
Array
The privileges that the role has on the source.
For more information, read Privileges.
users
id
String
The ID of the user.
Example 4deaf0e4-f05c-441c-bb37-d4d6e43fdec7
permissions
Array
The privileges that the user has on the source.
For more information, read Privileges.
owner
ownerId
String (UUID)
The unique identifier for the owner (user or role) of the source. This identifier corresponds to the user_id
or role_id
in the users
or roles
system table.
For more information, read the Users system table.
Example 5c9c394d-7395-4687-97dd-beac4384c359
ownerType
String
The type of the owner.
EnumUSER, ROLE
Adding a Source
Add a source.
Adding a SourcePOST /v0/projects/{project-id}/catalog
Parameters
project-id
path
String (UUID)
The unique identifier for the project that contains the source that you want to add.
Example 02d36975-73eb-47ed-9bb5-de73060380f6
entityType
body
String
Specifies the type of container. For sources, the type is source
.
body
Object
Configuration options for the specified data source.
type
body
String
Identifies the source type that you are configuring. The source type determines the config
parameters that display.
For more information, read Configuration Attributes.
EnumARCTIC, AWSGLUE, MSSQL, MYSQL, ORACLE, POSTGRES, REDSHIFT, S3, SNOWFLAKE
name
body
String
The user-defined name of the source.
Example customer-records
body
Source Metadata
The policies covering the update of a source’s metadata.
accelerationGracePeriodMs
body
Number
Identifies the length of time to keep reflections for all the datasets in a source before these reflections expire. The default setting is 10800000
milliseconds or three hours. The minimum amount of time that is supported is 3600000
milliseconds or one hour.
For more information, read Setting the Expiration Policy for Reflections.
Example 10800000
accelerationRefreshPeriodMs
body
Number
Identifies the refresh frequency for a dataset's reflections. The default setting is 3600000
milliseconds or one hour, which is also the minimum amount of time that is supported.
Example 3600000
body
Object
Object used to specify which users and roles should have privileges on the source and the specific privileges each user and role has. May include a users array, a roles array, or both. Not supported for Arctic sources and the primary Arctic catalog.
config
hostname
body
String
The name of the server you are connecting to.
Example 177.15.0.112
port
body
String
The port number of the server you are connecting to.
Example 3306
authenticationType
body
String
The type of authentication needed to connect to the server. Use ANONYMOUS when no authentication is needed and MASTER when credentials from a master database user is required.
EnumANONYMOUS, MASTER
netWriteTimeout
body
Number
The amount of time, in seconds, to wait for data from the server before aborting the connection.
Example 60
fetchSize
body
Number
The number of records to fetch per request. Set to 0
to enable Dremio to automatically set the size.
Example 200
maxIdleConns
body
Number
Sets the maximum number of idle connections that you want to keep.
Example 8
idleTimeSec
body
Number
Sets the idle time, in seconds, before a connection is evaluated for closure.
Example 60
propertyList
body
Array
An array of name/value pairs defining optional connection properties that is used by the source. Use a comma separated list to specify multiple name value pairs.
metadataPolicy
authTTLMs
body
Number
Sets the expiration for an authorization to a metadata policy, in milliseconds. The default time period is 24 hours (86400000
milliseconds). The minimum supported time period is one minute (60000
milliseconds).
Example 86400000
namesRefreshMs
body
Number
Sets when to run a refresh of a source, in milliseconds. The default time period is one hour (3600000
milliseconds). The minimum supported time period is one minute (60000
milliseconds).
Example 86400000
datasetRefreshAfterMs
body
Number
Determines how often the metadata in the dataset is refreshed, in milliseconds. The default refresh rate is one hour (3600000
milliseconds). The minimum supported refresh rate is one minute (60000
milliseconds).
Example 86400000
datasetExpireAfterMs
body
Number
Sets the amount of time, in milliseconds, to keep the metadata before it expires. The default time period is one hour (3600000
milliseconds). The minimum supported time period is one minute (60000
milliseconds).
Example 259200000
datasetUpdateMode
body
String
Sets the metadata policy for when a dataset is updated. Use PREFETCH_QUERIED
to update the details for previously queried objects in a source.
PREFETCH_QUERIED
deleteUnavailableDatasets
body
Boolean
Option to remove dataset definitions if the underlying data is unavailable to Dremio, for example, at the time the data is queried. The default setting is true
. Set to false
to keep the dataset definitions.
accessControlList
body
Array
The roles that should have privileges on the source and the privileges each role should have.
body
Array
The users that should have privileges on the source and the privileges each user should have.
roles
id
body
String
The ID of the role.
Example 9ab42853-bdef-465f-b9bb-381a13a9bf78
permissions
body
Array
The privileges that the role should have on the source.
For more information, read Privileges.
users
id
body
String
The ID of the user.
permissions
body
Array
The privileges that the user should have on the source.
For more information, read Privileges.
curl -X POST 'https://api.dremio.cloud/v0/projects/02d36975-73eb-47ed-9bb5-de73060380f6/catalog' \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json' \
-d '{
"entityType": "source",
"config": {
"hostname": "177.15.0.112",
"port": "3306",
"authenticationType": "ANONYMOUS",
"netWriteTimeout": 60,
"fetchSize": 200,
"maxIdleConns": 8,
"idleTimeSec": 60,
"propertyList": []
},
"type": "MYSQL",
"name": "recruitingdb",
"metadataPolicy": {
"authTTLMs": 86400000,
"namesRefreshMs": 86400000,
"datasetRefreshAfterMs": 86400000,
"datasetExpireAfterMs": 259200000,
"datasetUpdateMode": "PREFETCH_QUERIED",
"deleteUnavailableDatasets": true
},
"accelerationGracePeriodMs": 10800000,
"accelerationRefreshPeriodMs": 3600000,
"accessControlList": {
"roles": [
{
"id": "9ab42853-bdef-465f-b9bb-381a13a9bf78",
"permissions": []
}
],
"users": []
}
}'
{
"entityType": "source",
"config": {
"hostname": "177.15.0.112",
"port": "3306",
"authenticationType": "ANONYMOUS",
"netWriteTimeout": 60,
"fetchSize": 200,
"maxIdleConns": 8,
"idleTimeSec": 60,
"propertyList": []
},
"id": "629904cf-9c06-4ae6-8cc1-faf1d2f1ab5f",
"tag": "a927dd14-2c7d-4684-9de2-4000e05c0d07",
"type": "MYSQL",
"name": "recruitingdb",
"createdAt": "2022-09-12T07:22:19.334Z",
"metadataPolicy": {
"authTTLMs": 86400000,
"namesRefreshMs": 86400000,
"datasetRefreshAfterMs": 86400000,
"datasetExpireAfterMs": 259200000,
"datasetUpdateMode": "PREFETCH_QUERIED",
"deleteUnavailableDatasets": true
},
"accelerationGracePeriodMs": 10800000,
"accelerationRefreshPeriodMs": 3600000,
"accelerationNeverExpire": false,
"accelerationNeverRefresh": false,
"children": [
{
"id": "42312a90-1723-4bcb-b62c-bf9abee371b1",
"path": [
"recruitingdb",
"contacts"
],
"type": "CONTAINER",
"containerType": "FOLDER"
}
],
"allowCrossSourceSelection": false,
"disableMetadataValidityCheck": false,
"accessControlList": {
"roles": [
{
"id": "9ab42853-bdef-465f-b9bb-381a13a9bf78",
"permissions": []
}
],
"users": []
},
"permissions": [],
"checkTableAuthorizer": false,
"owner": {
"ownerId": "5c9c394d-7395-4687-97dd-beac4384c359",
"ownerType": "USER"
}
}
Responses
200
Source created successfully
400
Bad input parameter
401
Unauthorized access
Retrieving a Source by ID
Get information about the source using the source ID.
Retrieving a Source by IDGET /v0/projects/{project-id}/catalog/{id}
Parameters
project-id
path
String (UUID)
The unique identifier for the project that contains the source.
Example 02d36975-73eb-47ed-9bb5-de73060380f6
id
path
String (UUID)
The unique identifier for the source that you want to retrieve.
Example ffbe8c1d-1db7-48d1-9c58-f452838fedc0
include
query
String
Optional
When using include
, the name of a non-default response field is returned. You can retrieve either a list of access control list names - aclNames
- or a list of permissions
. Read Include and Exclude Query Parameters for usage examples.
aclNames, permissions
Example ffbe8c1d-1db7-48d1-9c58-f452838fedc0?include=permissions
exclude
query
String
Optional
When using exclude
, the children
field is excluded from the response. Read Include and Exclude Query Parameters for usage examples.
children
Example ffbe8c1d-1db7-48d1-9c58-f452838fedc0?exclude=children
curl -X GET 'https://api.dremio.cloud/v0/projects/02d36975-73eb-47ed-9bb5-de73060380f6/catalog/ffbe8c1d-1db7-48d1-9c58-f452838fedc0' \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json'
{
"entityType": "source",
"config": {
"hostname": "177.15.0.112",
"port": "3306",
"authenticationType": "ANONYMOUS",
"netWriteTimeout": 60,
"fetchSize": 200,
"maxIdleConns": 8,
"idleTimeSec": 60,
"propertyList": []
},
"id": "629904cf-9c06-4ae6-8cc1-faf1d2f1ab5f",
"tag": "a927dd14-2c7d-4684-9de2-4000e05c0d07",
"type": "MYSQL",
"name": "recruitingdb",
"createdAt": "2022-09-12T07:22:19.334Z",
"metadataPolicy": {
"authTTLMs": 86400000,
"namesRefreshMs": 86400000,
"datasetRefreshAfterMs": 86400000,
"datasetExpireAfterMs": 259200000,
"datasetUpdateMode": "PREFETCH_QUERIED",
"deleteUnavailableDatasets": true
},
"accelerationGracePeriodMs": 10800000,
"accelerationRefreshPeriodMs": 3600000,
"accelerationNeverExpire": false,
"accelerationNeverRefresh": false,
"children": [
{
"id": "42312a90-1723-4bcb-b62c-bf9abee371b1",
"path": [
"recruitingdb",
"contacts"
],
"type": "CONTAINER",
"containerType": "FOLDER"
}
],
"allowCrossSourceSelection": false,
"disableMetadataValidityCheck": false,
"accessControlList": {},
"permissions": [],
"checkTableAuthorizer": false,
"owner": {
"ownerId": "5c9c394d-7395-4687-97dd-beac4384c359",
"ownerType": "USER"
}
}
Responses
200
Source returned successfully
400
Bad input parameter
401
Unauthorized access
404
Not found
Retrieving a Source by Path
Get information about the source using the source's path.
Retrieving a Source by PathGET /v0/projects/{project-id}/catalog/by-path/{path}
Parameters
project-id
path
String (UUID)
The unique identifier for the project that contains the source.
Example 02d36975-73eb-47ed-9bb5-de73060380f6
path
path
String (UUID)
Source's location within Dremio, using forward slashes as separators. For example, for the source "Samples," the path is Samples
. If the name of any component in the path includes special characters for URLs, such as spaces, use URL encoding to replace the special characters with their UTF-8-equivalent characters. For example, "Dremio University" should be Dremio%20University
in the URL path.
Example recruitingdb
curl -X GET 'https://api.dremio.cloud/v0/projects/02d36975-73eb-47ed-9bb5-de73060380f6/catalog/by-path/recruitingdb' \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json'
{
"entityType": "source",
"config": {
"hostname": "177.15.0.112",
"port": "3306",
"authenticationType": "ANONYMOUS",
"netWriteTimeout": 60,
"fetchSize": 200,
"maxIdleConns": 8,
"idleTimeSec": 60,
"propertyList": []
},
"id": "629904cf-9c06-4ae6-8cc1-faf1d2f1ab5f",
"tag": "a927dd14-2c7d-4684-9de2-4000e05c0d07",
"type": "MYSQL",
"name": "recruitingdb",
"createdAt": "2022-09-12T07:22:19.334Z",
"metadataPolicy": {
"authTTLMs": 86400000,
"namesRefreshMs": 86400000,
"datasetRefreshAfterMs": 86400000,
"datasetExpireAfterMs": 259200000,
"datasetUpdateMode": "PREFETCH_QUERIED",
"deleteUnavailableDatasets": true
},
"accelerationGracePeriodMs": 10800000,
"accelerationRefreshPeriodMs": 3600000,
"accelerationNeverExpire": false,
"accelerationNeverRefresh": false,
"children": [
{
"id": "42312a90-1723-4bcb-b62c-bf9abee371b1",
"path": [
"recruitingdb",
"contacts"
],
"type": "CONTAINER",
"containerType": "FOLDER"
}
],
"allowCrossSourceSelection": false,
"disableMetadataValidityCheck": false,
"accessControlList": {},
"permissions": [],
"checkTableAuthorizer": false,
"owner": {
"ownerId": "5c9c394d-7395-4687-97dd-beac4384c359",
"ownerType": "USER"
}
}
Responses
200
Source returned successfully
400
Bad input parameter
401
Unauthorized access
404
Not found
Updating a Source
Edit the source information using the source ID.
Updating a SourcePUT /v0/projects/{project-id}/catalog/{id}
Parameters
project-id
path
String (UUID)
The unique identifier for the project that contains the source.
Example 02d36975-73eb-47ed-9bb5-de73060380f6
id
path
String (UUID)
The unique identifier for the source that you want to update.
Example ffbe8c1d-1db7-48d1-9c58-f452838fedc0
entityType
body
String
Specifies the type of container. For sources, the type is source
.
body
Object
Configuration options for the specified data source.
type
body
String
Identifies the source type that you are configuring. The source type determines the config
parameters that display.
For more information, read Configuration Attributes.
EnumARCTIC, AWSGLUE, MSSQL, MYSQL, ORACLE, POSTGRES, REDSHIFT, S3, SNOWFLAKE
name
body
String
The user-defined name of the source.
Example recruitingdb
body
Source Metadata
The policies covering the update of a source’s metadata.
accelerationGracePeriodMs
body
Number
Identifies the length of time to keep reflections for all the datasets in a source before these reflections expire. The default setting is 10800000
milliseconds or three hours. The minimum amount of time that is supported is 3600000
milliseconds or one hour.
For more information, read Setting the Expiration Policy for Reflections.
Example 10800000
accelerationRefreshPeriodMs
body
Number
Identifies the refresh frequency for a dataset's reflections. The default setting is 3600000
milliseconds or one hour, which is also the minimum amount of time that is supported.
Example 3600000
body
Object
Object used to specify which users and roles should have privileges on the source and the specific privileges each user and role has. May include a users array, a roles array, or both. Not supported for Arctic sources and the primary Arctic catalog.
config
hostname
body
String
The name of the server you are connecting to.
Example 177.15.0.112
port
body
String
The port number of the server you are connecting to.
Example 3306
authenticationType
body
String
The type of authentication needed to connect to the server. Use ANONYMOUS when no authentication is needed and MASTER when credentials from a master database user is required.
EnumANONYMOUS, MASTER
netWriteTimeout
body
Number
The amount of time, in seconds, to wait for data from the server before aborting the connection.
Example 60
fetchSize
body
Number
The number of records to fetch per request. Set to 0
to enable Dremio to automatically set the size.
Example 200
maxIdleConns
body
Number
Sets the maximum number of idle connections that you want to keep.
Example 8
idleTimeSec
body
Number
Sets the idle time, in seconds, before a connection is evaluated for closure.
Example 60
propertyList
body
Array
An array of name/value pairs defining optional connection properties that is used by the source. Use a comma separated list to specify multiple name value pairs.
metadataPolicy
authTTLMs
body
Number
Sets the expiration for an authorization to a metadata policy, in milliseconds. The default time period is 24 hours (86400000
milliseconds). The minimum supported time period is one minute (60000
milliseconds).
Example 86400000
namesRefreshMs
body
Number
Sets when to run a refresh of a source, in milliseconds. The default time period is one hour (3600000
milliseconds). The minimum supported time period is one minute (60000
milliseconds).
Example 86400000
datasetRefreshAfterMs
body
Number
Determines how often the metadata in the dataset is refreshed, in milliseconds. The default refresh rate is one hour (3600000
milliseconds). The minimum supported refresh rate is one minute (60000
milliseconds).
Example 86400000
datasetExpireAfterMs
body
Number
Sets the amount of time, in milliseconds, to keep the metadata before it expires. The default time period is one hour (3600000
milliseconds). The minimum supported time period is one minute (60000
milliseconds).
Example 259200000
datasetUpdateMode
body
String
Sets the metadata policy for when a dataset is updated. Use PREFETCH_QUERIED
to update the details for previously queried objects in a source.
PREFETCH_QUERIED
deleteUnavailableDatasets
body
Boolean
Option to remove dataset definitions if the underlying data is unavailable to Dremio, for example, at the time the data is queried. The default setting is true
. Set to false
to keep the dataset definitions.
accessControlList
body
Array
The roles that should have privileges on the source and the privileges each role should have.
body
Array
The users that should have privileges on the source and the privileges each user should have.
roles
id
body
String
The ID of the role.
Example 9ab42853-bdef-465f-b9bb-381a13a9bf78
permissions
body
Array
The privileges that the role should have on the source.
For more information, read Privileges.
users
id
body
String
The ID of the user.
permissions
body
Array
The privileges that the user should have on the source.
For more information, read Privileges.
curl -X GET 'https://api.dremio.cloud/v0/projects/02d36975-73eb-47ed-9bb5-de73060380f6/catalog/ffbe8c1d-1db7-48d1-9c58-f452838fedc0' \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json'
-d '{
"entityType": "source",
"config": {
"hostname": "177.15.0.112",
"port": "3306",
"authenticationType": "ANONYMOUS",
"netWriteTimeout": 60,
"fetchSize": 200,
"maxIdleConns": 8,
"idleTimeSec": 60,
"propertyList": []
},
"type": "MYSQL",
"name": "recruitingdb",
"metadataPolicy": {
"authTTLMs": 86400000,
"namesRefreshMs": 86400000,
"datasetRefreshAfterMs": 86400000,
"datasetExpireAfterMs": 259200000,
"datasetUpdateMode": "PREFETCH_QUERIED",
"deleteUnavailableDatasets": true
},
"accelerationGracePeriodMs": 10800000,
"accelerationRefreshPeriodMs": 3600000,
"accessControlList": {
"roles": [
{
"id": "9ab42853-bdef-465f-b9bb-381a13a9bf78",
"permissions": []
}
],
"users": []
}
}'
{
"entityType": "source",
"config": {
"hostname": "177.15.0.112",
"port": "3306",
"authenticationType": "ANONYMOUS",
"netWriteTimeout": 60,
"fetchSize": 200,
"maxIdleConns": 8,
"idleTimeSec": 60,
"propertyList": []
},
"id": "629904cf-9c06-4ae6-8cc1-faf1d2f1ab5f",
"tag": "a927dd14-2c7d-4684-9de2-4000e05c0d07",
"type": "MYSQL",
"name": "recruitingdb",
"createdAt": "2022-09-12T07:22:19.334Z",
"metadataPolicy": {
"authTTLMs": 86400000,
"namesRefreshMs": 86400000,
"datasetRefreshAfterMs": 86400000,
"datasetExpireAfterMs": 259200000,
"datasetUpdateMode": "PREFETCH_QUERIED",
"deleteUnavailableDatasets": true
},
"accelerationGracePeriodMs": 10800000,
"accelerationRefreshPeriodMs": 3600000,
"accelerationNeverExpire": false,
"accelerationNeverRefresh": false,
"children": [
{
"id": "42312a90-1723-4bcb-b62c-bf9abee371b1",
"path": [
"recruitingdb",
"contacts"
],
"type": "CONTAINER",
"containerType": "FOLDER"
}
],
"allowCrossSourceSelection": false,
"disableMetadataValidityCheck": false,
"accessControlList": {
"roles": [
{
"id": "9ab42853-bdef-465f-b9bb-381a13a9bf78",
"permissions": []
}
],
"users": []
},
"permissions": [],
"checkTableAuthorizer": false,
"owner": {
"ownerId": "5c9c394d-7395-4687-97dd-beac4384c359",
"ownerType": "USER"
}
}
Responses
204
No content
400
Bad input parameter
401
Unauthorized access
404
Not found
Deleting a Source
Delete a source.
Dremio does not support deleting the primary Arctic catalog.
DELETE /v0/projects/{project-id}/catalog/{id}
Parameters
project-id
path
String (UUID)
The unique identifier for the project that contains the source.
Example 02d36975-73eb-47ed-9bb5-de73060380f6
id
path
String (UUID)
The unique identifier for the source that you want to delete. Deletion is not supported for the primary Arctic catalog.
Example 21a74cb1-3fcf-48c6-8f69-94c257ec3da3
curl -X DELETE 'https://api.dremio.cloud/v0/projects/02d36975-73eb-47ed-9bb5-de73060380f6/catalog/21a74cb1-3fcf-48c6-8f69-94c257ec3da3' \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json'
No response
Responses
204
No content
400
Bad input parameter
401
Unauthorized access
404
Not found