Source

Type of the catalog object. For sources, the entityType is source.

Example source

Configuration settings for the source. The available parameters in the config object are different for different source types. For more information, read Source Configuration.

Example { "accessKey": "EXAMPLE78HT89VS4YJEL", "accessSecret": "$DREMIO_EXISTING_VALUE$", "secure": true, "rootPath": "/", "enableAsync": true, "compatibilityMode": false, "isCachingEnabled": true, "maxCacheSpacePct": 100, "requesterPays": false, "enableFileStatusCheck": true, "defaultCtasFormat": "ICEBERG", "isPartitionInferenceEnabled": false, "credentialType": "ACCESS_KEY" }

Unique identifier of the source.

Example 2b1be882-7012-4a99-8d6c-82e32e4562e4

Unique identifier of the version of the source. Dremio changes the tag whenever the source changes and uses the tag to ensure that PUT requests apply to the most recent version of the source.

Example T0/Zr1FOY3A=

Type of source.

Enum ADL , ADX , AMAZONELASTIC , AWSGLUE , AZURE_STORAGE , DB2 , DREMIOTODREMIO , ELASTIC , GCS , HDFS , HIVE , HIVE3 , MONGO , MSSQL , MYSQL , NAS , NESSIE , ORACLE , POSTGRES , REDSHIFT , S3 , SNOWFLAKE , SYNAPSE , TERADATA

Example S3

Name for the source.

Example AWS-S3_testgroup

Date and time that the source was created, in UTC format.

Example 2023-02-17T14:32:20.640Z

Information about the metadata policy for the source.

Example { "authTTLMs": 86400000, "namesRefreshMs": 3600000, "datasetRefreshAfterMs": 3600000, "datasetExpireAfterMs": 10800000, "datasetUpdateMode": "PREFETCH_QUERIED", "deleteUnavailableDatasets": true, "autoPromoteDatasets": false }

Maximum age allowed for reflection data used to accelerate queries on datasets in the source, in milliseconds. Default is 0. For more information, read Setting the Expiration Policy for Reflections.

Example 10800000

Refresh period for the data in all reflections on datasets in the source, in milliseconds. Default is 0.

Example 3600000

If reflections on datasets in the source never expire, set to true. Otherwise, set to false (default).

Example false

If reflections on datasets in the source never refresh, set to true. Otherwise, set to false (default).

Example false

Information about each catalog object in the source.

Example [ { "id": "dremio:/AWS-S3_testgroup/archive.dremio.com", "path": [ "AWS-S3_testgroup", "archive.dremio.com" ], "type": "CONTAINER", "containerType": "FOLDER" }, { "id": "dremio:/AWS-S3_testgroup/logs_east-1", "path": [ "AWS-S3_testgroup", "logs_east-1" ], "type": "CONTAINER", "containerType": "FOLDER" }, { "id": "dremio:/AWS-S3_testgroup/logs_west-1", "path": [ "AWS-S3_testgroup", "logs_west-1" ], "type": "CONTAINER", "containerType": "FOLDER" } ]

If the source is available for queries that can select from multiple sources, set to true. Otherwise, set to false (default).

Example false

To disable the check for expired metadata and require users to refresh manually, set to true. Otherwise, set to false (default).

Example false

Enterprise only. Information about users and roles with access to the source and the specific privileges each user or role has. May include an array of users, an array of roles, or both, depending on the configured access and privileges. The accessControlList array is empty if source-specific access control privileges are not set.

Example { "users": [ { "id": "ebe519ab-20e3-43ff-9b4c-b3ec590c7e65", "permissions": [ "VIEW_REFLECTION", "SELECT" ] } ], "roles": [ { "id": "c45ff4d8-e910-4f85-89db-9b8c29188a56", "permissions": [ "ALTER", "CREATE_TABLE", "DROP", "INSERT", "DELETE", "UPDATE", "TRUNCATE", "VIEW_REFLECTION", "ALTER_REFLECTION", "MODIFY", "MANAGE_GRANTS", "SELECT" ] } ] }

Enterprise-only. List of the privileges that you have on the source. Empty unless the request URL includes the permissions query parameter. For more information, read Privileges.

Example [ "READ", "WRITE", "ALTER_REFLECTION", "SELECT", "ALTER", "VIEW_REFLECTION", "MODIFY", "MANAGE_GRANTS", "CREATE_TABLE", "DROP", "EXTERNAL_QUERY", "INSERT", "TRUNCATE", "DELETE", "UPDATE", "EXECUTE", "CREATE_SOURCE", "ALL" ]

Not used. Has the value true.

Example true

Information about the source's owner.

Example { "ownerId": "4fb93af3-acc2-4b10-ad4b-64dd7070d365", "ownerType": "USER" }

Length of time to cache the privileges that the user has on the source, in milliseconds. For example, if authTTLMs is set to 28800000 (8 hours), Dremio checks the user's permission status once every 8 hours. Default is 86400000 (24 hours). Minimum is 60000 (1 minute).

Example 86400000

How often the source is refreshed, in milliseconds. Default is 3600000 (1 hour). Minimum is 60000 (1 minute).

Example 3600000

How often the metadata in the source's datasets is refreshed, in milliseconds. Default is 3600000 (1 hour). Minimum is 60000 (1 minute).

Example 3600000

Maximum age allowed for the metadata in the source's datasets, in milliseconds. Default is 3600000 (1 hour). Minimum is 60000 (1 minute).

Example 10800000

Approach Dremio uses for updating the metadata when updating datasets in the source.

PREFETCH: (deprecated) Dremio updates details for all datasets in a source.
PREFETCH_QUERIED: Dremio updates details for previously queried objects in a source.

Example PREFETCH_QUERIED

If Dremio removes dataset definitions from the source when the underlying data is unavailable, set to true (default). Otherwise, set to false.

Example true

If Dremio automatically formats files into tables when a user issues a query, set to true. Otherwise, set to false (default). Available only for datalake sources, such as Amazon S3 and Hive.

Example false

Unique identifier of the child catalog object. The ID can be a UUID like 1acab7b3-ee82-44c1-abcc-e86d56078d4d or a text path like dremio:/Samples/samples.dremio.com/zip_lookup.csv.

Example dremio:/AWS-S3_testgroup/archive.dremio.com

Path to the child catalog object within the source, expressed as an array. The path consists of the source, followed by the name of the folder, file, or dataset itself as the last item in the array.

Example [ "AWS-S3_testgroup", "archive.dremio.com" ]

Type of the catalog object.

Enum CONTAINER , FILE , DATASET

Example CONTAINER

For catalog objects with the type CONTAINER, the containerType is FOLDER.

Example FOLDER

For catalog objects with the type DATASET, the type of dataset. If the dataset is from an external source such as PostgreSQL, the datasetType is DIRECT. For tables, the datasetType is PROMOTED. For views, the datasetType is VIRTUAL.

Enum DIRECT , PROMOTED , VIRTUAL

Example VIRTUAL

Enterprise only. List of users with access to the source and the specific privileges each user has.

Example [ { "id": "ebe519ab-20e3-43ff-9b4c-b3ec590c7e65", "permissions": [ "VIEW_REFLECTION", "SELECT" ] } ]

Enterprise only. List of roles whose members have access to the source and the specific privileges each role has.

Example [ { "id": "c45ff4d8-e910-4f85-89db-9b8c29188a56", "permissions": [ "ALTER", "CREATE_TABLE", "DROP", "INSERT", "DELETE", "UPDATE", "TRUNCATE", "VIEW_REFLECTION", "ALTER_REFLECTION", "MODIFY", "MANAGE_GRANTS", "SELECT" ] } ]

Unique identifier of the source's owner.

Example 4fb93af3-acc2-4b10-ad4b-64dd7070d365

Type of owner of the source.

Enum USER , ROLE

Example USER

Enterprise only. Unique identifier of the user or role with access to the source.

Example ebe519ab-20e3-43ff-9b4c-b3ec590c7e65

Enterprise only. List of privileges the user or role has on the source. For more information, read Privileges.

Enum ALL , VIEW_REFLECTION , TRUNCATE , UPDATE , DELETE , DROP , MANAGE_GRANTS , EXTERNAL_QUERY , EXECUTE , ALTER , INSERT , MODIFY , SELECT , CREATE_SOURCE , WRITE , CREATE_TABLE , ALTER_REFLECTION , READ

Example [ "VIEW_REFLECTION", "SELECT" ] } ]

Type of the catalog object to create. For sources, the entityType is source.

Example source

Configuration settings for the source. The available parameters in the config object are different for different source types. For more information, read Source Configuration.

Example { "accessKey": "EXAMPLE78HT89VS4YJEL", "accessSecret": "EXAMPLEe3bcpKnAwgJ2WBpX8d9kEdhMz24guiR7L", "secure": true, "rootPath": "/", "enableAsync": true, "compatibilityMode": false, "isCachingEnabled": true, "maxCacheSpacePct": 100, "requesterPays": false, "enableFileStatusCheck": true, "defaultCtasFormat": "ICEBERG", "isPartitionInferenceEnabled": false, "credentialType": "ACCESS_KEY" }

Type of source to create.

Enum ADL , ADX , AMAZONELASTIC , AWSGLUE , AZURE_STORAGE , DB2 , DREMIOTODREMIO , ELASTIC , GCS , HDFS , HIVE , HIVE3 , MONGO , MSSQL , MYSQL , NAS , NESSIE , ORACLE , POSTGRES , REDSHIFT , S3 , SNOWFLAKE , SYNAPSE , TERADATA

Example S3

Name for the source.

Example AWS-S3_testgroup

Information about the metadata policy for the source.

Example { "authTTLMs": 86400000, "namesRefreshMs": 3600000, "datasetRefreshAfterMs": 3600000, "datasetExpireAfterMs": 10800000, "datasetUpdateMode": "PREFETCH_QUERIED", "deleteUnavailableDatasets": true, "autoPromoteDatasets": false }

Maximum age to allow for reflection data used to accelerate queries on datasets in the source, in milliseconds. Default is 0. For more information, read Setting the Expiration Policy for Reflections.

Example 10800000

Refresh period to use for the data in all reflections on datasets in the source, in milliseconds. Default is 0.

Example 3600000

If reflections on datasets in the source should never expire, set to true. Otherwise, set to false (default).

Example false

If reflections on datasets in the source should never refresh, set to true. Otherwise, set to false (default).

Example false

If the source should be available for queries that can select from multiple sources, set to true. Otherwise, set to false (default).

Example false

To disable the check for expired metadata and require users to refresh manually, set to true. Otherwise, set to false (default).

Example false

Enterprise only. Information about users and roles that should have access to the source and the specific privileges each user or role should have. May include an array of users, an array of roles, or both, depending on the configured access and privileges.

Example { "users": [ { "id": "ebe519ab-20e3-43ff-9b4c-b3ec590c7e65", "permissions": [ "VIEW_REFLECTION", "SELECT" ] } ], "roles": [ { "id": "c45ff4d8-e910-4f85-89db-9b8c29188a56", "permissions": [ "ALTER", "CREATE_TABLE", "DROP", "INSERT", "DELETE", "UPDATE", "TRUNCATE", "VIEW_REFLECTION", "ALTER_REFLECTION", "MODIFY", "MANAGE_GRANTS", "SELECT" ] } ] }

Length of time to cache the privileges that the user has on the source, in milliseconds. For example, if authTTLMs is set to 28800000 (8 hours), Dremio checks the user's permission status once every 8 hours. Default is 86400000 (24 hours). Minimum is 60000 (1 minute).

Example 86400000

How often the source should be refreshed, in milliseconds. Default is 3600000 (1 hour). Minimum is 60000 (1 minute).

Example 3600000

How often the metadata in the source's datasets should be refreshed, in milliseconds. Default is 3600000 (1 hour). Minimum is 60000 (1 minute).

Example 3600000

Maximum age to allow for the metadata in the source's datasets, in milliseconds. Default is 3600000 (1 hour). Minimum is 60000 (1 minute).

Example 10800000

Approach for Dremio to take for updating the metadata when updating datasets in the source.

PREFETCH: (deprecated) Dremio updates details for all datasets in a source.
PREFETCH_QUERIED: Dremio updates details for previously queried objects in a source.

Example PREFETCH_QUERIED

If Dremio should remove dataset definitions from the source when the underlying data is unavailable, set to true (default). Otherwise, set to false.

Example true

If Dremio should automatically format files into tables using default options when users issue queries, set to true. Otherwise, set to false (default). Available only for datalake sources, such as Amazon S3 and Hive.

Example false

Enterprise only. List of users who should have access to the source and the specific privileges each user should have.

Example [ { "id": "ebe519ab-20e3-43ff-9b4c-b3ec590c7e65", "permissions": [ "VIEW_REFLECTION", "SELECT" ] } ]

List of roles whose members should have access to the source and the specific privileges each role should have.

Example [ { "id": "c45ff4d8-e910-4f85-89db-9b8c29188a56", "permissions": [ "ALTER", "CREATE_TABLE", "DROP", "INSERT", "DELETE", "UPDATE", "TRUNCATE", "VIEW_REFLECTION", "ALTER_REFLECTION", "MODIFY", "MANAGE_GRANTS", "SELECT" ] } ]

Unique identifier of the user or role who should have access to the source.

Example ebe519ab-20e3-43ff-9b4c-b3ec590c7e65

List of privileges the user or role should have on the source. For more information, read Privileges.

Enum ALL , VIEW_REFLECTION , TRUNCATE , UPDATE , DELETE , DROP , MANAGE_GRANTS , EXTERNAL_QUERY , EXECUTE , ALTER , INSERT , MODIFY , SELECT , CREATE_SOURCE , WRITE , CREATE_TABLE , ALTER_REFLECTION , READ

Example [ "VIEW_REFLECTION", "SELECT" ] } ]

Status of the created source.

Enum good , bad , warn

Example good

Recommended action to take, if any, based on the status of the created source.

Status message, if any, for the created source.

Unique identifier of the source that you want to retrieve.

Example 2b1be882-7012-4a99-8d6c-82e32e4562e4

Include a non-default attribute in the response. The available value for the include query parameter is permissions. For more information, read include and exclude Query Parameters.

Example ?include=permissions

Exclude a default attribute from the response. The available value for the exclude query parameter is children. For more information, read include and exclude Query Parameters.

Example ?exclude=children

Name of the source that you want to retrieve. URI-encode the path to replace special characters in source names with their UTF-8 equivalents, such as %3A for a colon and %20 for a space.

Example AWS-S3_testgroup

Include a non-default attribute in the response. The available value for the include query parameter is permissions. For more information, read include and exclude Query Parameters.

Example ?include=permissions

Exclude a default attribute from the response. The available value for the exclude query parameter is children. For more information, read include and exclude Query Parameters.

Example ?exclude=children

Type of the catalog object to update. For sources, the entityType is source.

Example source

Configuration settings for the source. The available parameters in the config object are different for different source types. For more information, read Source Configuration.

Example { "accessKey": "EXAMPLE78HT89VS4YJEL", "accessSecret": "EXAMPLEe3bcpKnAwgJ2WBpX8d9kEdhMz24guiR7L", "secure": true, "rootPath": "/", "enableAsync": true, "compatibilityMode": false, "isCachingEnabled": true, "maxCacheSpacePct": 100, "requesterPays": false, "enableFileStatusCheck": true, "defaultCtasFormat": "ICEBERG", "isPartitionInferenceEnabled": false, "credentialType": "ACCESS_KEY" }

Unique identifier of the source.

Example 2b1be882-7012-4a99-8d6c-82e32e4562e4

Unique identifier of the version of the source that you want to update. Dremio uses the tag to ensure that you are requesting to update the most recent version of the source.

Example T0/Zr1FOY3A=

Type of the source that you want to update.

Enum ADL , ADX , AMAZONELASTIC , AWSGLUE , AZURE_STORAGE , DB2 , DREMIOTODREMIO , ELASTIC , GCS , HDFS , HIVE , HIVE3 , MONGO , MSSQL , MYSQL , NAS , NESSIE , ORACLE , POSTGRES , REDSHIFT , S3 , SNOWFLAKE , SYNAPSE , TERADATA

Example S3

Name of the source that you want to update.

Example AWS-S3_testgroup

Information about the metadata policy for the source.

Example { "authTTLMs": 86400000, "namesRefreshMs": 3600000, "datasetRefreshAfterMs": 3600000, "datasetExpireAfterMs": 10800000, "datasetUpdateMode": "PREFETCH_QUERIED", "deleteUnavailableDatasets": true, "autoPromoteDatasets": false }

Maximum age to allow for reflection data used to accelerate queries on datasets in the source, in milliseconds. For more information, read Setting the Expiration Policy for Reflections.

Example 10800000

Refresh period to use for the data in all reflections on datasets in the source, in milliseconds. Default is 0.

Example 3600000

If reflections on datasets in the source should never expire, set to true. Otherwise, set to false (default).

Example false

If reflections on datasets in the source should never refresh, set to true. Otherwise, set to false (default).

Example false

If the source should be available for queries that can select from multiple sources, set to true. Otherwise, set to false (default).

Example false

To disable the check for expired metadata and require users to refresh manually, set to true. Otherwise, set to false (default).

Example false

Enterprise only. Information about users and roles that should have access to the source and the specific privileges each user or role should have. May include an array of users, an array of roles, or both, depending on the configured access and privileges. To keep existing accessControlList settings while making other updates, duplicate the existing accessControlList object in the PUT request.

Example { "users": [ { "id": "ebe519ab-20e3-43ff-9b4c-b3ec590c7e65", "permissions": [ "VIEW_REFLECTION", "SELECT" ] } ], "roles": [ { "id": "c45ff4d8-e910-4f85-89db-9b8c29188a56", "permissions": [ "ALTER", "CREATE_TABLE", "DROP", "INSERT", "DELETE", "UPDATE", "TRUNCATE", "VIEW_REFLECTION", "ALTER_REFLECTION", "MODIFY", "MANAGE_GRANTS", "SELECT" ] } ] }

Length of time to cache the privileges that the user has on the source, in milliseconds. For example, if authTTLMs is set to 28800000 (8 hours), Dremio checks the user's privilege status once every 8 hours. Default is 86400000 (24 hours). Minimum is 60000 (1 minute).

Example 86400000

How often to refresh the source, in milliseconds. Default is 3600000 (1 hour). Minimum is 60000 (1 minute).

Example 3600000

How often to refresh the metadata in the source's datasets, in milliseconds. Default is 3600000 (1 hour). Minimum is 60000 (1 minute).

Example 3600000

Maximum age to allow for the metadata in the source's datasets, in milliseconds. Default is 3600000 (1 hour). Minimum is 60000 (1 minute).

Example 10800000

Approach for Dremio to take for updating the metadata when updating datasets in the source.

PREFETCH: (deprecated) Dremio updates details for all datasets in a source.
PREFETCH_QUERIED: Dremio updates details for previously queried objects in a source.

Example PREFETCH_QUERIED

If Dremio should remove dataset definitions from the source when the underlying data is unavailable, set to true (default). Otherwise, set to false.

Example true

If Dremio should automatically format files into tables when a user issues a query, set to true. Otherwise, set to false (default). Available only for datalake sources, such as Amazon S3 and Hive.

Example false

Enterprise only. List of users who should have access to the source and the specific privileges each user should have.

Example [ { "id": "ebe519ab-20e3-43ff-9b4c-b3ec590c7e65", "permissions": [ "VIEW_REFLECTION", "SELECT" ] } ]

Enterprise only. List of roles whose members should have access to the source and the specific privileges each role should have.

Example [ { "id": "c45ff4d8-e910-4f85-89db-9b8c29188a56", "permissions": [ "ALTER", "CREATE_TABLE", "DROP", "INSERT", "DELETE", "UPDATE", "TRUNCATE", "VIEW_REFLECTION", "ALTER_REFLECTION", "MODIFY", "MANAGE_GRANTS", "SELECT" ] } ]

Enterprise only. Unique identifier of the user or role who should have access to the source.

Example ebe519ab-20e3-43ff-9b4c-b3ec590c7e65

Enterprise only. List of privileges the user or role should have on the source. For more information, read Privileges.

Enum ALL , VIEW_REFLECTION , TRUNCATE , UPDATE , DELETE , DROP , MANAGE_GRANTS , EXTERNAL_QUERY , EXECUTE , ALTER , INSERT , MODIFY , SELECT , CREATE_SOURCE , WRITE , CREATE_TABLE , ALTER_REFLECTION , READ

Example [ "VIEW_REFLECTION", "SELECT" ] } ]

Unique identifier of the source that you want to delete.

Example 2b1be882-7012-4a99-8d6c-82e32e4562e4