Version: current [25.x]

Source Configuration

Dremio supports a variety of source types, and the available parameters of the config object differ among the source types. This page provides examples of the config object for each supported source type and descriptions of the config object's parameters that are available for each source type.

note

The source object contains the same parameters for all source types except for the parameters of the config object. Read the Source page for information about the parameters all source types have in common.

Amazon OpenSearch Service

Amazon OpenSearch Service Source config Object

{
  "config": {
    "scriptsEnabled": true,
    "showHiddenIndices": false,
    "showIdColumn": false,
    "readTimeoutMillis": 60000,
    "scrollTimeoutMillis": 300000,
    "usePainless": true,
    "scrollSize": 4000,
    "allowPushdownOnNormalizedOrAnalyzedFields": false,
    "pushdownWithKeyword": false,
    "warnOnRowCountMismatch": false,
    "encryptionValidationMode": "CERTIFICATE_AND_HOSTNAME_VALIDATION",
    "forceDoublePrecision": false,
    "hostname": "search-test.us-west-2.es.amazonaws.com",
    "port": 443,
    "authenticationType": "ACCESS_KEY",
    "accessKey": "AKIAQ3XZRGQRKEXAMPLE",
    "accessSecret": "$DREMIO_EXISTING_VALUE$",
    "overwriteRegion": true,
    "regionName": "US_WEST_1"
  }
}

Amazon OpenSearch Service config Parameters

scriptsEnabled Body Boolean Optional

If Dremio should use script pushdowns, set to true (default). Otherwise, set to false.

Example: true

showHiddenIndices Body Boolean Optional

To show hidden indices, set to true. Otherwise, set to false (default).

Example: false

showIdColumn Body Boolean Optional

To show the OpenSearch ID column, set to true. Otherwise, set to false (default).

Example: false

readTimeoutMillis Body Integer Optional

Time to wait to read data from the source, in milliseconds. Default is 60000.

Example: 60000

scrollTimeoutMillis Body Integer Optional

Time to wait for each scroll request, in milliseconds. Default is 300000.

Example: 300000

usePainless Body Boolean Optional

To use the Painless scripting language when connecting to Amazon OpenSearch Service, set to true (default). Otherwise, set to false.

Example: true

scrollSize Body Integer Optional

Scroll size for Dremio's Amazon OpenSearch Service requests. Default is 4000.

Example: 4000

allowPushdownOnNormalizedOrAnalyzedFields Body Boolean Optional

To enable pushdown filters and aggregations on analyzed text fields and normalized keyword fields, set to true. Otherwise, set to false (default). May produce unexpected results when enabled.

Example: false

pushdownWithKeyword Body Boolean Optional

If Dremio should perform keyword searches when pushing down fields mapped as text and keyword, set to true. Otherwise, set to false (default).

Example: false

warnOnRowCountMismatch Body Boolean Optional

If Dremio should warn the user when a query returns fewer Amazon OpenSearch Service records than expected instead of failing the query, set to true. Otherwise, set to false (default).

Example: false

encryptionValidationMode Body String Optional

Method to use to validate data encryption for the source.

Enum: CERTIFICATE_AND_HOSTNAME_VALIDATION, CERTIFICATE_ONLY_VALIDATION, NO_VALIDATION

Example: CERTIFICATE_AND_HOSTNAME_VALIDATION

forceDoublePrecision Body Boolean Optional

To force precision for double values, set to true. Otherwise, set to false (default).

Example: false

hostname Body String

Name of the host to use to connect to the Amazon OpenSearch Service source.

Example: search-test.us-west-2.es.amazonaws.com

port Body String

Port to use with the specified hostname to connect to the Amazon OpenSearch Service source. Default is 443.

Example: 443

authenticationType Body String

Type of authentication for Dremio to use to connect to the source.

Enum: NONE, EC2_METADATA, ACCESS_KEY, AWS_PROFILE

Example: ACCESS_KEY

accessKey Body String

AWS access key. Required for credentialType ACCESS_KEY.

Example: AKIAQ3XZRGQRKEXAMPLE

accessSecret Body String

The AWS access secret in plaintext or as the reference for the AWS access secret in Azure Key Vault, AWS Secrets Manager, or HashiCorp Vault. Required for credentialType ACCESS_KEY. To keep the access secret secure, Dremio returns the accessSecret value as $DREMIO_EXISTING_VALUE$ in API responses.

Plaintext accessSecret values must be prefixed with data:, — for example, data:,<plaintext_access_secret>. Plaintext accessSecret values that contain a colon (:) must be encoded in base64 format and prefixed with data:;base64, — for example, data:;base64,<base64_encoded_access_secret>.

overwriteRegion Body Boolean Optional

To override the default region for your AWS account with the region you specify with the regionName parameter, set to true. Otherwise, set to false (default).

Example: true

regionName Body String Optional

Name of the AWS region to use to overwrite the default region for your AWS account. Required if overwriteRegion is set to true.

Example: US_WEST_1

Amazon Redshift

Amazon Redshift Source config Object

{
  "config": {
    "connectionString": "jdbc:redshift://testing-redshift-cluster.r57mkjay4utq.us-west-2.redshift.amazonaws.com:5439/support",
    "username": "redshift-owner",
    "password": "$DREMIO_EXISTING_VALUE$",
    "authenticationType": "MASTER",
    "propertyList": [
      {
        "name": "ApplicationName",
        "value": "testingDashboard"
      }
    ],
    "fetchSize": 200,
    "maxIdleConns": 8,
    "idleTimeSec": 60,
    "queryTimeoutSec": 0
  }
}

Amazon Redshift config Parameters

connectionString Body String

JDBC connection string for the Redshift database.

Example: jdbc:redshift://testing-redshift-cluster.r57mkjay4utq.us-west-2.redshift.amazonaws.com:5439/support

username Body String

Username for authentication with master credentials or a secret resource URL.

Example: redshift-owner

password Body String

The password in plaintext or as the reference for the password in Azure Key Vault, AWS Secrets Manager, or HashiCorp Vault. To keep the password secure, Dremio returns the password value as $DREMIO_EXISTING_VALUE$ in API responses.

Plaintext password values must be prefixed with data:, — for example, data:,<plaintext_password>. Plaintext password values that contain a colon (:) must be encoded in base64 format and prefixed with data:;base64, — for example, data:;base64,<base64_encoded_password>.

secretResourceUrl Body String

For secret-based authentication, the Amazon Resource Name (ARN) for the secret. To keep the secret secure, Dremio returns the secretResourceUrl value as $DREMIO_EXISTING_VALUE$ in API responses.

Example: arn:aws:secretsmanager:us-west-2:123456789012㊙️my-rds-secret-VNenFy

authenticationType Body String

Type of authentication for Dremio to use to connect to the source.

ANONYMOUS: No authentication is needed.
MASTER: Use credentials from a master database user or use a secret resource URL.
AWS_PROFILE: Use an AWS source.

Example: MASTER

propertyList Body Array of Object Optional

Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.

Example: [{"name": "ApplicationName","value": "testingDashboard"}]

fetchSize Body Integer Optional

Number of records to fetch at one time. Default is 200. If set to 0, Dremio automatically decides how many records to fetch.

Example: 200

maxIdleConns Body Integer Optional

Maximum number of connections that can be idle at any given time. Default is 8.

Example: 8

idleTimeSec Body Integer Optional

Maximum time that a connection can remain idle before Dremio terminates it, in seconds. Default is 60.

Example: 60

queryTimeoutSec Body Integer

Maximum time to allow for query execution, in seconds. When the query timeout expires, the connection returns to an idle state. Default is 0.

Example: 0

Parameters of Objects in the `propertyList` Array

name Body String Optional

Name of the connection property.

Example: ApplicationName

value Body String Optional

Value for the connection property.

Example: testingDashboard

Amazon S3

Amazon S3 Source config Object

{
  "config": {
    "accessKey": "EXAMPLE78HT89VS4YJEL",
    "accessSecret": "$DREMIO_EXISTING_VALUE$",
    "secure": true,
    "externalBucketList": [
      "external_bucket_1",
      "external_bucket_2"
    ],
    "propertyList": [
      {
        "name": "fs.s3a.proxy.host",
        "value": "proxyHost.example.com"
      }
    ],
    "rootPath": "/",
    "enableAsync": true,
    "compatibilityMode": false,
    "isCachingEnabled": true,
    "maxCacheSpacePct": 100,
    "whitelistedBuckets": [
      "archive.dremio.com",
      "logs_east-1",
      "logs_west-1"
    ],
    "requesterPays": false,
    "enableFileStatusCheck": true,
    "defaultCtasFormat": "ICEBERG",
    "isPartitionInferenceEnabled": false,
    "credentialType": "ACCESS_KEY"
  }
}

Amazon S3 config Parameters

accessKey Body String

AWS access key. Required for credentialType ACCESS_KEY.

Example: EXAMPLE78HT89VS4YJEL

accessSecret Body String

assumedRoleARN Body String Optional

Amazon Resource Name (ARN) for the AWS Identity and Access Management (IAM) role to assume. Optional for credentialType ACCESS_KEY and EC2_METADATA. Not applicable for credentialType AWS_PROFILE and NONE.

Example: arn:aws:iam::594632595346:role/OrganizationAccountAccessRole

awsProfile Body String

AWS profile name. If you do not provide a profile name, Dremio uses the default profile.

Example: example-profile

secure Body Boolean Optional

If set to enable a secure connection with SSL encryption between the Amazon S3 bucket and Dremio, the value is true (default). Otherwise, the value is false.

Example: true

externalBucketList Body Array of String Optional

List of external buckets that are not included with the specified AWS account credentials.

Example: ["external_bucket_1","external_bucket_2"]

propertyList Body Array of Object Optional

Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.

Example: [{"name": "fs.s3a.proxy.host","value": "proxyHost.example.com"}]

rootPath Body String Optional

Root path of the Amazon S3 bucket. Default is /.

Example: /

enableAsync Body Boolean Optional

To enable asynchronous access for the source and allow cloud caching to support simultaneous actions such as adding and editing, set to true (default). Otherwise, set to false.

Example: true

compatibilityMode Body Boolean Optional

To enable the use of Amazon S3-compatible storage, set to true. Otherwise, set to false (default).

Example: false

isCachingEnabled Body Boolean Optional

To enable local caching, set to true (default). Otherwise, set to false.

Example: true

maxCacheSpacePct Body Integer Optional

Maximum percentage of the total available cache space to use on any single executor node. Used only when isCachingEnabled is set to true. Minimum is 1. Maximum is 100 (default).

Example: 100

whitelistedBuckets Body Array of String Optional

List of approved S3 buckets to include in the source. Buckets entered must be valid. Misspelled or non-existent buckets will not appear in the resulting source. Omit to include all buckets that are available in the source. If you omit the whitelistedBuckets array in a PUT request, Dremio updates the source to include all available buckets. To keep existing approved buckets while making other updates, duplicate the existing whitelistedBuckets array in the PUT request.

Example: ["archive.dremio.com","logs_east-1","logs_west-1"]

requesterPays Body Boolean Optional

If the requester (instead of the bucket owner) pays the cost of the S3 request and the data downloaded from the Amazon S3 bucket, set to true. Otherwise, set to false (default).

Example: false

enableFileStatusCheck Body Boolean Optional

To allow Dremio to check whether a file exists in the Amazon S3 bucket before handling errors gracefully, set to true (default). Otherwise, set to false. If no files are missing from the Amazon S3 bucket or a file's access permissions have not changed, set to false to reduce communication with the Amazon S3 bucket.

Example: true

defaultCtasFormat Body String Optional

Default format for the tables you create in Dremio. Default is ICEBERG.

Enum: ICEBERG, PARQUET

Example: ICEBERG

isPartitionInferenceEnabled Body Boolean Optional

To enable partition column inference, set to true. Otherwise, set to false (default). For more information, read Partition Column Inference.

Example: false

credentialType Body String Optional

Type of credential for Dremio to use to connect to the source.

Enum: NONE, EC2_METADATA, ACCESS_KEY, AWS_PROFILE

Example: ACCESS_KEY

Parameters of Objects in the `propertyList` Array

name Body String Optional

Name of the connection property.

Example: fs.s3a.proxy.host

value Body String Optional

Value for the connection property.

Example: proxyHost.example.com

AWS Glue Data Catalog

AWS Glue Data Catalog Source config Object

{
  "config": {
    "regionNameSelection": "US_EAST_1",
    "accessKey": "EXAMPLE78HT89VS4YJEL",
    "accessSecret": "$DREMIO_EXISTING_VALUE$",
    "secure": true,
    "enableAsync": true,
    "isCachingEnabled": true,
    "maxCacheSpacePct": 100,
    "assumedRoleARN": "arn:aws:iam::594632595346:role/OrganizationAccountAccessRole",
    "lakeFormationEnableAccessPermissions": false,
    "defaultCtasFormat": "ICEBERG",
    "propertyList": [
      {
        "name": "hive.metastore.warehouse.dir",
        "value": "s3a://example.com/buckets/glue_location"
      }
    ],
    "credentialType": "ACCESS_KEY",
    "allowedDatabases": ["sales","finance"]
  }
}

Amazon Glue Data Catalog config Parameters

regionNameSelection Body String

AWS region where the Glue Data Catalog is located.

Example: US_EAST_1

accessKey Body String

AWS access key.

Example: EXAMPLE78HT89VS4YJEL

accessSecret Body String

awsProfile Body String

AWS profile name. If you do not provide a profile name, Dremio uses the default profile.

Example: example-profile

secure Body Boolean Optional

To enable a secure connection with SSL encryption between the Glue Data Catalog and Dremio, set to true (default). Otherwise, set to false.

Example: true

enableAsync Body Boolean Optional

To enable asynchronous access for the source and allow cloud caching to support simultaneous actions such as adding and editing, set to true (default). Otherwise, set to false.

Example: true

isCachingEnabled Body Boolean Optional

To enable local caching, set to true (default). Otherwise, set to false.

Example: true

maxCacheSpacePct Body Integer Optional

Maximum percentage of the total available cache space to use on any single executor node. Used only when isCachingEnabled is set to true. Minimum is 1. Maximum is 100 (default).

Example: 100

assumedRoleARN Body String Optional

Example: arn:aws:iam::594632595346:role/OrganizationAccountAccessRole

lakeFormationEnableAccessPermissions Body Boolean

To enforce Lake Formation access permissions on datasets so that Dremio confirms whether users have the required permissions to perform queries, set to true. Otherwise, set to false (default).

Example: false

defaultCtasFormat Body String Optional

Default format for the tables you create in Dremio. Default is ICEBERG.

Enum: ICEBERG, PARQUET

Example: ICEBERG

propertyList Body Array of Object Optional

Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.

Example: [{"name": "hive.metastore.warehouse.dir","value": "s3a://example.com/buckets/glue_location"}]

credentialType Body String Optional

Type of credential for Dremio to use to connect to the source.

Enum: NONE, EC2_METADATA, ACCESS_KEY, AWS_PROFILE

Example: ACCESS_KEY

allowedDatabases Body Array of String

A list of the databases that users of Dremio are allowed to access. Present the names in a comma-separated list. Databases entered must be valid. Misspelled or non-existent databases will not appear in the resulting source.

If one or more databases are added or removed from the list, the change takes effect immediately.

Example: ["sales","finance”]

Parameters of Objects in the `propertyList` Array

name Body String Optional

Name of the connection property.

Example: hive.metastore.warehouse.dir

value Body String Optional

Value for the connection property.

Example: s3a://example.com/buckets/glue_location

Azure Storage

Azure Storage Source config Object

{
  "config": {
    "accountKind": "STORAGE_V2",
    "accountName": "azurestoragev2accountname",
    "accessKey": "$DREMIO_EXISTING_VALUE$",
    "rootPath": "/",
    "enableSSL": true,
    "enableAsync": true,
    "credentialsType": "ACCESS_KEY",
    "isCachingEnabled": true,
    "maxCacheSpacePct": 100,
    "defaultCtasFormat": "ICEBERG",
    "propertyList": [
      {
        "name": "dremio.azure.mode",
        "value": "STORAGE_V2"
      }
    ],
    "isPartitionInferenceEnabled": false
  }
}

Azure Storage config Parameters

accountKind Body String

Type of Azure Storage account.

Enum: STORAGE_V1, STORAGE_V2

Example: STORAGE_V2

accountName Body String

Name of the Azure Storage account.

Example: azurestoragev2accountname

accessKey Body String

The Azure access key in plaintext or as the reference for the Azure access key in Azure Key Vault, AWS Secrets Manager, or HashiCorp Vault. Required for credentialsType ACCESS_KEY. To keep the Azure access key secure, Dremio returns the accessKey value as $DREMIO_EXISTING_VALUE$ in API responses.

Plaintext accessKey values must be prefixed with data:, — for example, data:,<plaintext_access_key>. Plaintext accessKey values that contain a colon (:) must be encoded in base64 format and prefixed with data:;base64, — for example, data:;base64,<base64_encoded_access_key>.

clientSecret Body String

The Azure client secret in plaintext or as the reference for the Azure client secret in Azure Key Vault, AWS Secrets Manager, or HashiCorp Vault. Include only for credentialsType AZURE_ACTIVE_DIRECTORY. To keep the Azure client secret secure, Dremio returns the clientSecret value as $DREMIO_EXISTING_VALUE$ in API responses.

Plaintext clientSecret values must be prefixed with data:, — for example, data:,<plaintext_client_secret>. Plaintext clientSecret values that contain a colon (:) must be encoded in base64 format and prefixed with data:;base64, — for example, data:;base64,<base64_encoded_client_secret>.

rootPath Body String Optional

Root path of the Azure Storage source. Default is /.

Example: /

enableSSL Body Boolean Optional

To enable a secure connection with SSL encryption between the Azure Storage source and Dremio, set to true (default). Otherwise, set to false.

Example: true

enableAsync Body Boolean Optional

To enable asynchronous access for the source and allow cloud caching to support simultaneous actions such as adding and editing, set to true (default). Otherwise, set to false.

Example: true

credentialType Body String Optional

Type of credential for Dremio to use to connect to the source.

Enum: ACCESS_KEY, AZURE_ACTIVE_DIRECTORY

Example: ACCESS_KEY

isCachingEnabled Body Boolean Optional

To enable local caching, set to true (default). Otherwise, set to false.

Example: true

maxCacheSpacePct Body Integer Optional

Maximum percentage of the total available cache space to use on any single executor node. Used only when isCachingEnabled is set to true. Minimum is 1. Maximum is 100 (default).

Example: 100

defaultCtasFormat Body String Optional

Default format for the tables you create in Dremio. Default is ICEBERG.

Enum: ICEBERG, PARQUET

Example: ICEBERG

propertyList Body Array of Object Optional

Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.

Example: [{"name": "dremio.azure.mode","value": "STORAGE_V2"}]

isPartitionInferenceEnabled Body Boolean Optional

To enable partition column inference, set to true. Otherwise, set to false (default). For more information, read Partition Column Inference.

Example: false

Parameters of Objects in the `propertyList` Array

name Body String Optional

Name of the connection property.

Example: dremio.azure.mode

value Body String Optional

Value for the connection property.

Example: STORAGE_V2

Dremio-to-Dremio Connector

Dremio-to-Dremio Connector Source config Object

{
  "config": {
    "hostType": "DIRECT",
    "hostname": "35.219.202.153",
    "port": "31010",
    "username": "exampleuser",
    "password": "$DREMIO_EXISTING_VALUE$",
    "useSsl": false,
    "userImpersonation": false,
    "maxIdleConns": 8,
    "idleTimeSec": 60,
    "queryTimeoutSec": 0,
    "propertyList": [
      {
        "name": "disableCertificateVerification",
        "value": "false"
      }
    ]
  }
}

Dremio-to-Dremio Connector config Parameters

hostType Body String

Connection method to use. To connect directly to a coordinator node of the cluster, DIRECT. To connect to an external ZooKeeper instance that coordinates the nodes of the cluster, ZOOKEEPER.

Enum: DIRECT, ZOOKEEPER

Example: DIRECT

hostname Body String

Hostname of the coordinator node or ZooKeeper instance.

Example: 35.219.202.153

port Body String

Port number of the coordinator node or ZooKeeper instance. Default is 31010.

Example: 31010

username Body String

Username for authentication.

Example: exampleuser

password Body String

useSsl Body Boolean Optional

If the source cluster is configured to use TLS for connections, set to true. Otherwise, set to false.

Example: false

userImpersonation Body Boolean Optional

To allow users to run queries on the source cluster under their own user IDs rather than the user ID for the account used to authenticate, set to true. Otherwise, set to false (default). If set to true, the source cluster must be configured to allow inbound impersonation. For more information, read Inbound Impersonation.

Example: false

maxIdleConns Body Integer Optional

Maximum number of connections that can be idle at any given time. Default is 8.

Example: 8

idleTimeSec Body Integer Optional

Maximum time that a connection can remain idle before Dremio terminates it, in seconds. Default is 60.

Example: 60

queryTimeoutSec Body Integer Optional

Maximum time to allow for query execution, in seconds. When the query timeout expires, the connection returns to an idle state. Default is 0.

Example: 0

propertyList Body Array of Object Optional

Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.

Example: [{"name": "disableCertificateVerification","value": "false"}]

Parameters of Objects in the `propertyList` Array

name Body String Optional

Name of the connection property.

Example: disableCertificateVerification

value Body String Optional

Value for the connection property.

Example: false

Elasticsearch

Elasticsearch Source config Object

{
  "config": {
    "scriptsEnabled": true,
    "showHiddenIndices": false,
    "showIdColumn": false,
    "readTimeoutMillis": 60000,
    "scrollTimeoutMillis": 300000,
    "usePainless": true,
    "scrollSize": 4000,
    "allowPushdownOnNormalizedOrAnalyzedFields": false,
    "warnOnRowCountMismatch": false,
    "encryptionValidationMode": "CERTIFICATE_AND_HOSTNAME_VALIDATION",
    "forceDoublePrecision": false,
    "hostList": [
      {
        "hostname": "172.25.0.208",
        "port": 9200
      }
    ],
    "authenticationType": "ANONYMOUS",
    "sslEnabled": false,
    "useWhitelist": false
  }
}

Elasticsearch config Parameters

scriptsEnabled Body Boolean Optional

If Dremio should use script pushdowns, set to true (default). Otherwise, set to false.

Example: true

showHiddenIndices Body Boolean Optional

To show hidden indices, set to true. Otherwise, set to false (default).

Example: false

showIdColumn Body Boolean Optional

To show the elasticsearch ID column, set to true. Otherwise, set to false (default).

Example: false

readTimeoutMillis Body Integer Optional

Time to wait to read data from the source, in milliseconds. Default is 60000.

Example: 60000

scrollTimeoutMillis Body Integer Optional

Time to wait for each scroll request, in milliseconds. Default is 300000.

Example: 300000

usePainless Body Boolean Optional

To use the Painless scripting language when connecting to Elasticsearch 5.0+, set to true (default). Otherwise, set to false.

Example: true

scrollSize Body Integer Optional

Scroll size for Dremio's Elasticsearch requests. Default is 4000. The scrollSize value must be less than or equal to the setting for index.max_result_window in Elasticsearch.

Example: 4000

allowPushdownOnNormalizedOrAnalyzedFields Body Boolean Optional

To enable pushdown filters and aggregations on analyzed text fields and normalized keyword fields, set to true. Otherwise, set to false (default). May produce unexpected results when enabled.

Example: false

warnOnRowCountMismatch Body Boolean Optional

If Dremio should warn the user when a query returns fewer Elasticsearch records than expected instead of failing the query, set to true. Otherwise, set to false (default).

Example: false

encryptionValidationMode Body String Optional

Method to use to validate data encryption for the source.

Enum: CERTIFICATE_AND_HOSTNAME_VALIDATION, CERTIFICATE_ONLY_VALIDATION, NO_VALIDATION

Example: CERTIFICATE_AND_HOSTNAME_VALIDATION

forceDoublePrecision Body Boolean Optional

To force precision for double values, set to true. Otherwise, set to false (default).

Example: false

hostList Body Array of Object

Information about Elasticsearch hosts. Each object in the hostList includes the hostname and the corresponding port for the host.

Example: [{"hostname": "172.25.0.208","port": 9200}]

authenticationType Body String

Type of authentication for Dremio to use to connect to the source.

Enum: ANONYMOUS, MASTER

Example: ANONYMOUS

username Body String

Username for authenticating with master credentials. Required for authenticationType MASTER.

Example: exampleuser

password Body String

The password in plaintext or as the reference for the password in Azure Key Vault, AWS Secrets Manager, or HashiCorp Vault. Required for authenticationType MASTER. To keep the password secure, Dremio returns the password value as $DREMIO_EXISTING_VALUE$ in API responses.

sslEnabled Body Boolean Optional

To enable a secure connection with SSL encryption between Elasticsearch and Dremio, set to true. Otherwise, set to false (default).

Example: false

useWhitelist Body Boolean Optional

To query only the hosts specified in the hostList, set to true. Otherwise, set to false (default).

Example: false

Parameters of Objects in the `hostList` Array

hostname Body String Optional

Name of the host to use to connect to the Elasticsearch source.

Example: 172.25.0.208

port Body Integer Optional

Port to use with the specified hostname to connect to the Elasticsearch source. Default is 9200.

Example: 9200

Google Cloud Storage

Google Cloud Storage Source config Object

{
  "config": {
    "projectId": "dremio-4321",
    "authMode": "AUTO",
    "rootPath": "/",
    "asyncEnabled": true,
    "cachingEnable": true,
    "cachePercent": 70,
    "privateKeyId": "",
    "clientEmail": "",
    "clientId": "",
    "defaultCtasFormat": "ICEBERG",
    "propertyList": [
      {
        "name": "dremio.gcs.clientEmail",
        "value": "exampleuser@dremio-project.iam.gserviceaccount.com"
      }
    ],
    "isPartitionInferenceEnabled": false
  }
}

Google Cloud Storage config Parameters

projectId Body String

Project ID for the Google Cloud Storage project.

Example: dremio-4321

authMode Body String

Type of authentication for Dremio to use to connect to the source.

Enum: AUTO, SERVICE_ACCOUNT_KEYS

Example: AUTO

rootPath Body String Optional

Root path of the Google Cloud Storage source. Default is /.

Example: /

asyncEnabled Body Boolean Optional

To enable asynchronous access for the source and allow cloud caching to support simultaneous actions such as adding and editing, set to true (default). Otherwise, set to false.

Example: true

cachingEnable Body Boolean Optional

To enable local caching, set to true (default). Otherwise, set to false.

Example: true

cachePercent Body Integer Optional

Maximum percentage of the total available cache space to use on any single executor node. Used only when cachingEnable is set to true. Default is 70. Minimum is 1. Maximum is 100.

Example: 70

privateKeyId Body String

Service account key ID for the Google Cloud Storage service account. Required for authMode SERVICE_ACCOUNT_KEYS.

Example: f302b86d0247be19393bb66ea142887fc9621360

privateKey Body String

The service account key in plaintext or as the reference for the service account key in Azure Key Vault, AWS Secrets Manager, or HashiCorp Vault. Required for authMode SERVICE_ACCOUNT_KEYS. To keep the service account key secure, Dremio returns the privateKey value as $DREMIO_EXISTING_VALUE$ in API responses.

Plaintext privateKey values must be prefixed with data:, — for example, data:,<plaintext_service_account_key>. Plaintext privateKey values that contain a colon (:) must be encoded in base64 format and prefixed with data:;base64, — for example, data:;base64,<base64_encoded_service_account_key>.

clientEmail Body String

Email address associated with the Google Cloud Storage service account. Required only if authenticating with the Service Account Keys method.

Example: exampleuser@dremio-project.iam.gserviceaccount.com

clientId Body String

Client ID for the key pair. Required only if authenticating with the Service Account Keys method.

Example: 243769857672272684657

defaultCtasFormat Body String Optional

Default format for the tables you create in Dremio. Default is ICEBERG.

Enum: ICEBERG, PARQUET

Example: ICEBERG

propertyList Body Array of Object Optional

Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.

Example: [{"name": "dremio.gcs.clientEmail","value": "exampleuser@dremio-project.iam.gserviceaccount.com"}]

isPartitionInferenceEnabled Body Boolean Optional

To enable partition column inference, set to true. Otherwise, set to false (default). For more information, read Partition Column Inference.

Example: false

Parameters of Objects in the `propertyList` Array

name Body String Optional

Name of the connection property.

Example: dremio.gcs.clientEmail

value Body String Optional

Value for the connection property.

Example: exampleuser@dremio-project.iam.gserviceaccount.com

Hadoop Distributed File System (HDFS)

HDFS Source config Object

{
  "config": {
    "hostname": "172.23.0.208",
    "port": 8020,
    "enableImpersonation": false,
    "rootPath": "/",
    "shortCircuitFlag": "SYSTEM",
    "enableAsync": true,
    "isCachingEnabled": false,
    "maxCacheSpacePct": 100,
    "defaultCtasFormat": "ICEBERG",
    "isPartitionInferenceEnabled": false,
    "impersonationUserDelegationMode": "AS_IS",
    "propertyList": [
      {
        "name": "dfs.client.socket-timeout",
        "value": "120000"
      }
    ],
    "vdsAccessDelegationEnabled": true
  }
}

HDFS config Parameters

hostname Body String

HDFS NameNode hostname to use to connect to the HDFS source.

Example: 172.23.0.208

port Body Integer

HDFS NameNode port to use to connect to the HDFS source. Default is 8020.

Example: 8020

enableImpersonation Body Boolean Optional

To enable user-specific file access permissions by turning on impersonation in HDFS sources, set to true. Otherwise, set to false (default).

Example: false

rootPath Body String Optional

Root path of the HDFS source. Default is /.

Example: /

shortCircuitFlag Body String Optional

Implementation status for short-circuit local reads. Default is SYSTEM.

Enum: ENABLED, DISABLED, SYSTEM

Example: SYSTEM

shortCircuitSocketPath Body String Optional

Socket path to use, if shortCircuitFlag is enabled.

Example: /var/lib/hadoop-hdfs/dn-socket

enableAsync Body Boolean Optional

To enable asynchronous access for the source and allow cloud caching to support simultaneous actions such as adding and editing, set to true (default). Otherwise, set to false.

Example: true

isCachingEnabled Body Boolean Optional

To enable local caching, set to true. Otherwise, set to false (default).

Example: false

maxCacheSpacePct Body Integer Optional

Maximum percentage of the total available cache space to use on any single executor node. Used only when cachingEnable is set to true. Minimum is 1. Maximum is 100 (default).

Example: 100

defaultCtasFormat Body String Optional

Default format for the tables you create in Dremio. Default is ICEBERG.

Enum: ICEBERG, PARQUET

Example: ICEBERG

isPartitionInferenceEnabled Body Boolean Optional

To enable partition column inference, set to true. Otherwise, set to false (default). For more information, read Partition Column Inference.

Example: false

impersonationUserDelegationMode Body String Optional

Capitalization method to use for impersonation usernames. Default is AS_IS.

Enum: AS_IS, LOWERCASE, UPPERCASE

Example: AS_IS

propertyList Body Array of Object Optional

Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.

Example: [{"name": "dfs.client.socket-timeout","value": "120000"}]

vdsAccessDelegationEnabled Body Boolean Optional

To use the view owner as the impersonated username when enableImpersonation is enabled, set to true (default). To use the user who submits the query as the impersonated username, set to false.

Example: true

Parameters of Objects in the `propertyList` Array

name Body String Optional

Name of the connection property.

Example: dfs.client.socket-timeout

value Body String Optional

Value for the connection property.

Example: 120000

Hive 2.x

Hive 2.x Source config Object

{
  "config": {
    "hostname": "172.23.0.208",
    "port": 9083,
    "enableSasl": false,
    "propertyList": [
      {
        "name": "hive.server2.enable.doAs",
        "value": "false"
      }
    ],
    "secretPropertyList": [
      {
        "name": "fs.s3a.secret.key",
        "value": "70SPup32UsIZaA6c2n6bf3rQONTD6Zn6OqvcUhBy"
      }
    ],
    "enableAsync": true,
    "isCachingEnabledForS3AndAzureStorage": true,
    "isCachingEnabledForHDFS": true,
    "maxCacheSpacePct": 100,
    "defaultCtasFormat": "ICEBERG",
    "authType": "STORAGE",
    "rangerServiceName": "",
    "rangerHostURL": "",
    "impersonationUserDelegationMode": "AS_IS",
    "vdsAccessDelegationEnabled": true
  }
}

Hive 2.x config Parameters

hostname Body String

IP address for the Hive metastore host.

Example: 172.23.0.208

port Body Integer

Port to use with the specified hostname to connect to the Hive source. Default is 9083.

Example: 9083

enableSasl Body Boolean Optional

To enable Simple Authentication and Security Layer (SASL), set to true (default). Otherwise, set to false.

Example: false

kerberosPrincipal Body String Optional

Name of the Kerberos principal identity for SASL. Required only if enableSasl is true.

Example: primary/instance@REALM

propertyList Body Array of Object Optional

Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.

Example: [{"name": "hive.server2.enable.doAs","value": "false"}]

secretPropertyList Body Array of Object Optional

Additional connection properties for the source. The values are kept masked in the Credentials section of the Advanced Options page of the Source Settings dialog in the Dremio UI. Each object includes the name of the property and the corresponding value to use and to keep secret.

Example: [{"name": "fs.s3a.secret.key","value": "70SPup32UsIZaA6c2n6bf3rQONTD6Zn6OqvcUhBy"}]

enableAsync Body Boolean Optional

To enable asynchronous access for the source and allow cloud caching to support simultaneous actions such as adding and editing, set to true (default). Otherwise, set to false.

Example: true

isCachingEnabledForS3AndAzureStorage Body Boolean Optional

To enable local caching for Amazon S3 and Azure Storage, set to true (default). Otherwise, set to false.

Example: true

isCachingEnabledForHDFS Body Boolean Optional

To enable local caching for Hadoop Distributed File System (HDFS), set to true. Otherwise, set to false (default).

Example: false

maxCacheSpacePct Body Integer Optional

Maximum percentage of the total available cache space to use on any single executor node. Minimum is 1. Maximum is 100 (default).

Example: 100

defaultCtasFormat Body String Optional

Default format for the tables you create in Dremio. Default is ICEBERG.

Enum: ICEBERG, PARQUET

Example: ICEBERG

authType Body Boolean Optional

Type of authentication for Dremio to use to connect to the source. Default is STORAGE.

Enum: STORAGE, SQL, RANGER

Example: STORAGE

rangerServiceName Body String Optional

The Ranger service name, if you are using Ranger-based authentication.

Example: hivedev

rangerHostURL Body String Optional

The Ranger host URL, if you are using Ranger-based authentication.

Example: http://example.com/6080

impersonationUserDelegationMode Body String Optional

Capitalization method to use for impersonation usernames. Default is AS_IS.

Enum: AS_IS, LOWERCASE, UPPERCASE

Example: AS_IS

vdsAccessDelegationEnabled Body Boolean Optional

To use the view owner as the impersonated username when enableImpersonation is enabled, set to true (default). To use the user who submits the query as the impersonated username, set to false.

Example: true

Parameters of Objects in the `propertyList` Array

name Body String Optional

Name of the connection property.

Example: hive.server2.enable.doAs

value Body String Optional

Value for the connection property.

Example: false

Parameters of Objects in the `secretPropertyList` Array

name Body String Optional

Name of the connection property.

Example: fs.s3a.secret.key

value Body String Optional

Value for the connection property. The value is masked in the Dremio UI.

Example: 70SPup32UsIZaA6c2n6bf3rQONTD6Zn6OqvcUhBy

Hive 3.x

Hive 3.x Source config Object

{
  "config": {
    "hostname": "172.23.0.177",
    "port": 9084,
    "enableSasl": false,
    "propertyList": [
      {
        "name": "fs.s3a.aws.credentials.provider",
        "value": "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"
      }
    ],
    "secretPropertyList": [
      {
        "name": "fs.s3a.secret.key",
        "value": "70SPup32UsIZaA6c2n6bf3rQONTD6Zn6OqvcUhBy"
      }
    ],
    "enableAsync": true,
    "isCachingEnabledForS3AndAzureStorage": true,
    "isCachingEnabledForHDFS": false,
    "maxCacheSpacePct": 100,
    "defaultCtasFormat": "ICEBERG",
    "authType": "STORAGE",
    "rangerServiceName": "",
    "rangerHostURL": "",
    "impersonationUserDelegationMode": "AS_IS",
    "vdsAccessDelegationEnabled": true
  }
}

Hive 3.x config Parameters

hostname Body String

IP address for the Hive metastore host.

Example: 172.23.0.177

port Body Integer

Port to use with the specified hostname to connect to the Hive source. Default is 9083.

Example: 9084

enableSasl Body Boolean Optional

To enable Simple Authentication and Security Layer (SASL), set to true (default). Otherwise, set to false.

Example: false

kerberosPrincipal Body String Optional

Name of the Kerberos principal identity for SASL. Required only if enableSasl is true.

Example: primary/instance@REALM

propertyList Body Array of Object Optional

Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.

Example: [{"name": "fs.s3a.aws.credentials.provider","value": "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"}]

secretPropertyList Body Array of Object Optional

Example: [{"name": "fs.s3a.secret.key","value": "70SPup32UsIZaA6c2n6bf3rQONTD6Zn6OqvcUhBy"}]

enableAsync Body Boolean Optional

To enable asynchronous access for the source and allow cloud caching to support simultaneous actions such as adding and editing, set to true (default). Otherwise, set to false.

Example: true

isCachingEnabledForS3AndAzureStorage Body Boolean Optional

To enable local caching for Amazon S3 and Azure Storage, set to true (default). Otherwise, set to false.

Example: true

isCachingEnabledForHDFS Body Boolean Optional

To enable local caching for Hadoop Distributed File System (HDFS), set to true. Otherwise, set to false (default).

Example: false

maxCacheSpacePct Body Integer Optional

Maximum percentage of the total available cache space to use on any single executor node. Minimum is 1. Maximum is 100 (default).

Example: 100

defaultCtasFormat Body String Optional

Default format for the tables you create in Dremio. Default is ICEBERG.

Enum: ICEBERG, PARQUET

Example: ICEBERG

authType Body Boolean Optional

Type of authentication for Dremio to use to connect to the source. Default is STORAGE.

Enum: STORAGE, SQL, RANGER

Example: STORAGE

rangerServiceName Body String Optional

The Ranger service name, if you are using Ranger-based authentication.

Example: hivedev

rangerHostURL Body String Optional

The Ranger host URL, if you are using Ranger-based authentication.

Example: http://example.com/6080

impersonationUserDelegationMode Body String Optional

Capitalization method to use for impersonation usernames. Default is AS_IS.

Enum: AS_IS, LOWERCASE, UPPERCASE

Example: AS_IS

vdsAccessDelegationEnabled Body Boolean Optional

To use the view owner as the impersonated username when enableImpersonation is enabled, set to true (default). To use the user who submits the query as the impersonated username, set to false.

Example: true

Parameters of Objects in the `propertyList` Array

name Body String Optional

Name of the connection property.

Example: fs.s3a.aws.credentials.provider

value Body String Optional

Value for the connection property.

Example: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider

Parameters of Objects in the `secretPropertyList` Array

name Body String Optional

Name of the connection property.

Example: fs.s3a.secret.key

value Body String Optional

Value for the connection property. The value is masked in the Dremio UI.

Example: 70SPup32UsIZaA6c2n6bf3rQONTD6Zn6OqvcUhBy

IBM Db2

IBM Db2 Source config Object

{
  "config": {
    "database": "tpch",
    "hostname": "172.25.1.245",
    "username": "exampleuser",
    "password": "$DREMIO_EXISTING_VALUE$",
    "port": "50000",
    "fetchSize": 200,
    "maxIdleConns": 8,
    "idleTimeSec": 60,
    "queryTimeoutSec": 0,
    "propertyList": [
      {
        "name": "deferPrepares",
        "value": "true"
      }
    ]
  }
}

IBM Db2 config Parameters

database Body String

IBM Db2 database for Dremio to use.

Example: tpch

hostname Body String

IBM Db2 hostname.

Example: 172.25.1.245

username Body String

Username for authentication.

Example: exampleuser

password Body String

port Body String

IBM Db2 port number. Default is 50000.

Example: 50000

fetchSize Body Integer Optional

Number of records to fetch at one time. Default is 200. If set to 0, Dremio automatically decides how many records to fetch.

Example: 200

maxIdleConns Body Integer Optional

Maximum number of connections that can be idle at any given time. Default is 8.

Example: 8

idleTimeSec Body Integer Optional

Maximum time that a connection can remain idle before Dremio terminates it, in seconds. Default is 60.

Example: 60

queryTimeoutSec Body Integer Optional

Maximum time to allow for query execution, in seconds. When the query timeout expires, the connection returns to an idle state. Default is 0.

Example: 0

propertyList Body Array of Object Optional

Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.

Example: [{"name": "deferPrepares","value": "true"}]

Parameters of Objects in the `propertyList` Array

name Body String Optional

Name of the connection property.

Example: deferPrepares

value Body String Optional

Value for the connection property.

Example: true

MapR File System (MapR FS)

MapR FS Source config Object

{
  "config": {
    "clusterName": "my.cluster.com",
    "enableImpersonation": false,
    "secure": false,
    "rootPath": "/",
    "defaultCtasFormat": "ICEBERG",
    "isPartitionInferenceEnabled": false,
    "impersonationUserDelegationMode": "AS_IS",
    "vdsAccessDelegationEnabled": true
  }
}

MapR FS config Parameters

clusterName Body String

Name of the MapR FS cluster.

Example: my.cluster.com

enableImpersonation Body Boolean Optional

To enable user-specific file access permissions by turning on impersonation in MapR FS sources, set to true. Otherwise, set to false (default).

Example: false

secure Body Boolean Optional

To enable a secure connection with SSL encryption between MapR FS and Dremio, set to true. Otherwise, set to false (default).

Example: false

rootPath Body String Optional

Root path of the MapR FS source. Default is /.

Example: /

defaultCtasFormat Body String Optional

Default format for the tables you create in Dremio. Default is ICEBERG.

Enum: ICEBERG, PARQUET

Example: ICEBERG

isPartitionInferenceEnabled Body Boolean Optional

To enable partition column inference, set to true. Otherwise, set to false (default). For more information, read Partition Column Inference.

Example: false

impersonationUserDelegationMode Body String Optional

Capitalization method to use for impersonation usernames. Default is AS_IS.

Enum: AS_IS, LOWERCASE, UPPERCASE

Example: AS_IS

vdsAccessDelegationEnabled Body Boolean Optional

To use the view owner as the impersonated username when enableImpersonation is enabled, set to true (default). To use the user who submits the query as the impersonated username, set to false.

Example: true

Microsoft Azure Data Explorer

Microsoft Azure Data Explorer Source config Object

{
  "config": {
    "clusterUri": "https://prod.eastus.kusto.windows.net",
    "tenantId": "6e334262-b0c6-4c36-9faf-380e97e42964",
    "appId": "c3e4554f-514a-4912-8b75-93800f0d6c71",
    "appSecret": "$DREMIO_EXISTING_VALUE$",
    "databaseName": "datatypes",
    "maxIdleConns": 8,
    "idleTimeSec": 60,
    "queryTimeoutSec": 0
  }
}

Microsoft Azure Date Explorer config Parameters

clusterUri Body String

URI for the Microsoft Entra ID tenant used by Dremio.

Example: https://prod.eastus.kusto.windows.net

tenantId Body String (UUID)

Unique identifier of the Microsoft Entra ID tenant.

Example: 6e334262-b0c6-4c36-9faf-380e97e42964

appId Body String (UUID)

Unique identifier of the Microsoft Entra ID application used when Dremio connects

Example: c3e4554f-514a-4912-8b75-93800f0d6c71

appSecret Body String

The application secret in plaintext or as the reference for the application secret in Azure Key Vault, AWS Secrets Manager, or HashiCorp Vault. To keep the application secret secure, Dremio returns the appSecret value as $DREMIO_EXISTING_VALUE$ in API responses.

Plaintext appSecret values must be prefixed with data:, — for example, data:,<plaintext_application_secret>. Plaintext appSecret values that contain a colon (:) must be encoded in base64 format and prefixed with data:;base64, — for example, data:;base64,<base64_encoded_application_secret>.

databaseName Body String

Default Azure Data Explorer database for Dremio to use.

Example: datatypes

maxIdleConns Body Integer Optional

Maximum number of connections that can be idle at any given time. Default is 8.

Example: 8

idleTimeSec Body Integer Optional

Maximum time that a connection can remain idle before Dremio terminates it, in seconds. Default is 60.

Example: 60

queryTimeoutSec Body Integer Optional

Maximum time to allow for query execution, in seconds. When the query timeout expires, the connection returns to an idle state. Default is 0.

Example: 0

Microsoft Azure Synapse Analytics

Microsoft Azure Synapse Analytics Source config Object

{
  "config": {
    "hostname": "synapsedata.sql.azuresynapse.net",
    "port": "1433",
    "username": "exampleuser",
    "password": "$DREMIO_EXISTING_VALUE$",
    "authenticationType": "MASTER",
    "fetchSize": 200,
    "useSsl": false,
    "enableServerVerification": true,
    "maxIdleConns": 8,
    "idleTimeSec": 60,
    "queryTimeoutSec": 0,
    "database": "dedicatedpool",
    "propertyList": [
      {
        "name": "applicationIntent",
        "value": "ReadWrite"
      }
    ]
  }
}

Microsoft Azure Synapse Analytics config Parameters

hostname Body String

Microsoft Azure Synapse Analytics hostname.

Example: synapsedata.sql.azuresynapse.net

port Body String Optional

Microsoft Azure Synapse Analytics port number. Default is 1433.

Example: 1433

username Body String

Username for authentication. Required for authenticationType MASTER.

Example: exampleuser

password Body String

authenticationType Body String

Type of authentication to use to allow Dremio to connect to the Microsoft Azure Synapse Analytics source.

ANONYMOUS: No authentication is needed.
MASTER: Use credentials from a master database user.

Example: MASTER

fetchSize Body Integer Optional

Number of records to fetch at one time. Default is 200. If set to 0, Dremio automatically decides how many records to fetch.

Example: 200

useSsl Body Boolean Optional

If the Microsoft Azure Synapse Analytics source is configured to use TLS for connections, set to true. Otherwise, set to false (default).

Example: false

enableServerVerification Body Boolean Optional

To verify the certificate for the Microsoft Azure Synapse Analytics source when connecting, set to true. Otherwise, set to false (default).

Example: false

maxIdleConns Body Integer Optional

Maximum number of connections that can be idle at any given time. Default is 8.

Example: 8

idleTimeSec Body Integer Optional

Maximum time that a connection can remain idle before Dremio terminates it, in seconds. Default is 60.

Example: 60

queryTimeoutSec Body Integer Optional

Maximum time to allow for query execution, in seconds. When the query timeout expires, the connection returns to an idle state. Default is 0.

Example: 0

database Body String Optional

Default Microsoft Azure Synapse Analytics database for Dremio to use.

Example: dedicatedpool

propertyList Body Array of Object Optional

Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.

Example: [{"name": "applicationIntent","value": "ReadWrite"}]

Parameters of Objects in the `propertyList` Array

name Body String Optional

Name of the connection property.

Example: applicationIntent

value Body String Optional

Value for the connection property.

Example: ReadWrite

Microsoft SQL Server

Microsoft SQL Server Source config Object

{
  "config": {
    "hostname": "172.25.0.10",
    "port": "1433",
    "username": "tpch",
    "password": "$DREMIO_EXISTING_VALUE$",
    "authenticationType": "MASTER",
    "fetchSize": 200,
    "useSsl": false,
    "enableServerVerification": true,
    "maxIdleConns": 8,
    "idleTimeSec": 60,
    "queryTimeoutSec": 0,
    "showOnlyConnectionDatabase": false,
    "propertyList": [
      {
        "name": "applicationIntent",
        "value": "ReadWrite"
      }
    ]
  }
}

Microsoft SQL Server config Parameters

hostname Body String

Microsoft SQL Server hostname.

Example: 172.25.0.10

port Body String

Microsoft SQL Server port number. Default is 5432.

Example: 1433

username Body String

Username for authentication. Required for authenticationType MASTER.

Example: tpch

password Body String

authenticationType Body String

Type of authentication to use to allow Dremio to connect to the Microsoft SQL Server source.

ANONYMOUS: No authentication is needed.
MASTER: Use credentials from a master database user.

Example: MASTER

fetchSize Body Integer Optional

Number of records to fetch at one time. Default is 200. If set to 0, Dremio automatically decides how many records to fetch.

Example: 200

useSsl Body Boolean Optional

If the Microsoft SQL Server source is configured to use TLS for connections, set to true. Otherwise, set to false (default).

Example: false

enableServerVerification Body Boolean Optional

To verify the certificate for the Microsoft SQL Server source when connecting, set to true. Otherwise, set to false (default).

Example: false

maxIdleConns Body Integer Optional

Maximum number of connections that can be idle at any given time. Default is 8.

Example: 8

idleTimeSec Body Integer Optional

Maximum time that a connection can remain idle before Dremio terminates it, in seconds. Default is 60.

Example: 60

queryTimeoutSec Body Integer Optional

Maximum time to allow for query execution, in seconds. When the query timeout expires, the connection returns to an idle state. Default is 0.

Example: 0

showOnlyConnectionDatabase Body Boolean Optional

To hide all databases that the credentialed user can access except the specified connection database, set to true. Otherwise, set to false (default).

Example: false

propertyList Body Array of Object Optional

Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.

Example: [{"name": "applicationIntent","value": "ReadWrite"}]

Parameters of Objects in the `propertyList` Array

name Body String Optional

Name of the connection property.

Example: applicationIntent

value Body String Optional

Value for the connection property.

Example: ReadWrite

MongoDB

MongoDB Source config Object

{
  "config": {
    "hostList": [
      {
        "hostname": "172.23.0.208",
        "port": 27017
      }
    ],
    "useSsl": false,
    "authenticationType": "ANONYMOUS",
    "authDatabase": "admin",
    "authenticationTimeoutMillis": 2000,
    "secondaryReadsOnly": false,
    "subpartitionSize": 0,
    "sampleSize": 4095,
    "sampleMethod": "FIRST",
    "propertyList": [
      {
        "name": "maxPoolSize",
        "value": "100"
      }
    ],
    "useCaseInsensitiveFieldNames": false
  }
}

MongoDB config Parameters

hostList Body Array of Object

Information about MongoDB hosts. Each object in the hostList includes the hostname and the corresponding port for the host.

Example: [{"hostname": "172.23.0.208","port": 27017}]

useSsl Body Boolean Optional

To force a secure connection with SSL encryption between MongoDB and Dremio, set to true. Otherwise, set to false (default).

Example: false

authenticationType Body String

Type of authentication for Dremio to use to connect to the source.

Enum: ANONYMOUS, MASTER

Example: ANONYMOUS

username Body String

Username for authentication. Required for authenticationType MASTER.

Example: exampleuser

password Body String

authDatabase Body String

Name of the MongoDB database to authenticate against.

Example: admin

authenticationTimeoutMillis Body Integer Optional

Time to wait for authentication requests, in milliseconds. Default is 2000.

Example: 2000

secondaryReadsOnly Body Boolean Optional

To disable reading from primaries, set to true. Otherwise, set to false (default). Enabling secondaryReadsOnly may degrade performance.

Example: false

subpartitionSize Body String Optional

Number of records to read by query fragments. Use subpartitionSize to increase query parallelism. Default is 0.

Example: 0

sampleSize Body Integer Optional

Number of records to read when sampling to determine the schema for a collection. Default is 4095. If the value is 0, the sample size is unlimited.

Example: 4095

sampleMethod Body String Optional

Method by which records should be read when sampling a collection to determine the schema. Default is FIRST.

Enum: FIRST, LAST

Example: FIRST

propertyList Body Array of Object Optional

Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.

Example: [{"name": "maxPoolSize","value": "100"}]

useCaseInsensitiveFieldNames Body Boolean Optional

To treat field names as case-insensitive, set to true. Otherwise, set to false (default). If set to true, Dremio records all known variations of a field name when learning the schema and uses them when pushing an operation down to MongoDB.

Example: false

Parameters of Objects in the `hostList` Array

hostname Body String

Name of the host to use to connect to the MongoDB source. If MongoDB is sharded, use the mongos host. Otherwise, use the mongod host.

Example: 172.23.0.208

port Body Integer

Port to use with the specified hostname to connect to the MongoDB source. Default is 27017.

Example: 27017

Parameters of Objects in the `propertyList` Array

name Body String Optional

Name of the connection property.

Example: maxPoolSize

value Body String Optional

Value for the connection property.

Example: 100

MySQL

MySQL Source config Object

{
  "config": {
    "hostname": "rdbms-linux-20231031.c.company-4321.internal",
    "port": "3306",
    "username": "exampleuser",
    "password": "$DREMIO_EXISTING_VALUE$",
    "authenticationType": "MASTER",
    "fetchSize": 200,
    "netWriteTimeout": 60,
    "maxIdleConns": 8,
    "idleTimeSec": 60,
    "propertyList": [
      {
        "name": "DataSource",
        "value": "sales2021"
      }
    ],
    "queryTimeoutSec": 0
  }
}

mySQL config Parameters

hostname Body String

Name of the host to use to connect to the mySQL source.

Example: rdbms-linux-20221031.c.dremio-4321.internal

port Body Integer

Port to use with the specified hostname to connect to the mySQL source. Default is 3306.

Example: 3306

username Body String

Username for authentication. Required for authenticationType MASTER.

Example: exampleuser

password Body String

authenticationType Body String

Type of authentication for Dremio to use to connect to the source.

Enum: ANONYMOUS, MASTER

Example: MASTER

fetchSize Body Integer Optional

Number of records to fetch at one time. Default is 200. If set to 0, Dremio automatically decides how many records to fetch.

Example: 200

netWriteTimeout Body Integer Optional

Length of time, in seconds, before an idle connection is eligible to be closed. Default is 60.

Example: 60

maxIdleConns Body Integer Optional

Maximum number of connections that can be idle at any given time. Default is 8.

Example: 8

idleTimeSec Body Integer Optional

Maximum time that a connection can remain idle before Dremio terminates it, in seconds. Default is 60.

Example: 60

propertyList Body Array of Object Optional

Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.

Example: [{"name": "DataSource","value": "sales2021"}]

queryTimeoutSec Body Integer Optional

Maximum time to allow for query execution, in seconds. When the query timeout expires, the connection returns to an idle state. Default is 0.

Example: 0

Parameters of Objects in the `propertyList` Array

name Body String Optional

Name of the connection property.

Example: DataSource

value Body String Optional

Value for the connection property.

Example: sales2021

Network-attached Storage (NAS)

NAS Source config Object

{
  "config": {
    "path": "/home/dremio/",
    "defaultCtasFormat": "ICEBERG",
    "isPartitionInferenceEnabled": false
  }
}

NAS config Parameters

path Body String

Path on the filesystem to use as the root for the source. The path must be accessible on all nodes.

Example: /home/dremio/

defaultCtasFormat Body String Optional

Default format for the tables you create in Dremio. Default is ICEBERG.

Enum: ICEBERG, PARQUET

Example: ICEBERG

isPartitionInferenceEnabled Body Boolean Optional

To enable partition column inference, set to true. Otherwise, set to false (default). For more information, read Partition Column Inference.

Example: false

Nessie

Nessie Source config Object

{
  "config": {
    "nessieEndpoint": "http://172.25.0.138:19120/api/v2",
    "nessieAuthType": "BEARER",
    "nessieAccessToken": "$DREMIO_EXISTING_VALUE$",
    "asyncEnabled": true,
    "isCachingEnabled": true,
    "maxCacheSpacePct": 100,
    "credentialType": "ACCESS_KEY",
    "awsAccessKey": "AKIAQ3XZRGQRKEXAMPLE",
    "awsAccessSecret": "$DREMIO_EXISTING_VALUE$",
    "awsRootPath": "/tmp.dremio.com/myFolder",
    "propertyList": [
      {
        "name": "fs.s3a.proxy.host",
        "value": "proxyHost.example.com"
      }
    ],
    "secure": true
  }
}

Nessie config Parameters

nessieEndpoint Body String

IP address and port that you have set up for your Nessie server.

Example: http://172.25.0.138:19120/api/v2

nessieAuthType Body String Optional

Type of Nessie authentication for Dremio to use to connect to the source. If set to NONE, Nessie authentication is not enforced on the Nessie server and other Dremio users can view the source without authenticating. If set to BEARER, Nessie authentication is enforced with an OpenID bearer token (default).

Enum: NONE, BEARER

Example: BEARER

nessieAccessToken Body String Optional

The bearer access token in plaintext or as the reference for the bearer access token in Azure Key Vault, AWS Secrets Manager, or HashiCorp Vault. Required for nessieAuthType BEARER. To keep the bearer access token secure, Dremio returns the nessieAccessToken value as $DREMIO_EXISTING_VALUE$ in API responses.

Plaintext nessieAccessToken values must be prefixed with data:, — for example, data:,<plaintext_nessie_access_token>. Plaintext nessieAccessToken values that contain a colon (:) must be encoded in base64 format and prefixed with data:;base64, — for example, data:;base64,<base64_encoded_nessie_access_token>.

asyncEnabled Body Boolean Optional

To enable asynchronous access for the source and allow cloud caching to support simultaneous actions such as adding and editing, set to true (default). Otherwise, set to false.

Example: true

isCachingEnabled Body Boolean Optional

To enable local caching, set to true (default). Otherwise, set to false.

Example: true

maxCacheSpacePct Body Integer Optional

Maximum percentage of the total available cache space to use on any single executor node. Used only when isCachingEnabled is set to true. Minimum is 1. Maximum is 100 (default).

Example: 100

storageType Body String

The storage provider that you are using for your Nessie source.

Enum: AWS, AZURE, GCS

Example: AWS

awsRootPath Body String Optional

The Amazon S3 path prefix that Dremio should use as the write location for new Iceberg metadata and data. The root path includes the name of the Amazon S3 bucket, followed by the names of any folders. Required if you are using Amazon S3 as the storage provider for your Nessie source.

Example: /tmp.dremio.com/myFolder

credentialType Body String Optional

Type of credential for Dremio to use to connect to AWS for storage of metadata and data files for tables and views. Required if you are using Amazon S3 as the storage provider for your Nessie source.

Enum: NONE, EC2_METADATA, ACCESS_KEY, AWS_PROFILE

Example: ACCESS_KEY

awsAccessKey Body String Optional

AWS access key. Required if you are using credentialType ACCESS_KEY and Amazon S3 as the storage provider for your Nessie source.

Example: AKIAQ3XZRGQRKEXAMPLE

awsAccessSecret Body String Optional

The AWS access secret in plaintext or as the reference for the AWS access secret in Azure Key Vault, AWS Secrets Manager, or HashiCorp Vault. Required for credentialType ACCESS_KEY. To keep the access secret secure, Dremio returns the awsAccessSecret value as $DREMIO_EXISTING_VALUE$ in API responses.

Plaintext awsAccessSecret values must be prefixed with data:, — for example, data:,<plaintext_access_secret>. Plaintext awsAccessSecret values that contain a colon (:) must be encoded in base64 format and prefixed with data:;base64, — for example, data:;base64,<base64_encoded_access_secret>.

assumedRoleARN Body String Optional

Example: arn:aws:iam::594632595346:role/OrganizationAccountAccessRole

awsProfile Body String Optional

The AWS profile name. Applies only if you are using the credentialType AWS_PROFILE and Amazon S3 as the storage provider for your Nessie source. If you do not provide a profile name, Dremio uses the default profile.

Example: example-profile

azureStorageAccount Body String Optional

The name of the Azure Storage account to use. Required if you are using Azure Storage as the storage provider for your Nessie source.

Example: prodstorage12

azureRootPath Body String Optional

The path in your Azure Storage account that Dremio should use as the write location for new Iceberg metadata and data. The root path includes the name of the Azure Storage container, followed by the names of any folders. Required if you are using Azure Storage as the storage provider for your Nessie source.

Example: /containername/folder1/folder2

azureAuthenticationType Body String Optional

The method you want to use to authenticate to Azure Storage: access key (ACCESS_KEY) or Microsoft Entra ID (AZURE_ACTIVE_DIRECTORY). If you choose ACCESS_KEY, you must also include the azureAccessKey parameter in your request. If you choose AZURE_ACTIVE_DIRECTORY, you must also include the azureApplicationId, azureClientSecret, and azureOAuthTokenEndpoint parameters in your request. Required if you are using Azure Storage as the storage provider for your Nessie source.

Enum: ACCESS_KEY, AZURE_ACTIVE_DIRECTORY

Example: ACCESS_KEY

azureAccessKey Body String Optional

The shared access key for the Azure Storage account. Required if you are using the azureAuthenticationType ACCESS_KEY and Azure Storage as the storage provider for your Nessie source.

Example: abc123DEF456ghi789JKL012mno345PQR678stu901vwx234yzA567bcd890EFG123hij45EXAMPLE=

azureApplicationId Body String Optional

The ID for the application (client) in Azure. Required if you are using the azureAuthenticationType AZURE_ACTIVE_DIRECTORY and Azure Storage as the storage provider for your Nessie source.

Example: 0d416550-e530-45ce-8367-e4aa4EXAMPLE

azureClientSecret Body String Optional

The Azure client secret in plaintext or as the reference for the Azure client secret in Azure Key Vault, AWS Secrets Manager, or HashiCorp Vault. Required if you are using the azureAuthenticationType AZURE_ACTIVE_DIRECTORY and Azure Storage as the storage provider for your Nessie source. To keep the client secret secure, Dremio returns the azureClientSecret value as $DREMIO_EXISTING_VALUE$ in API responses.

Plaintext azureClientSecret values must be prefixed with data:, — for example, data:,<plaintext_client_secret>. Plaintext azureClientSecret values that contain a colon (:) must be encoded in base64 format and prefixed with data:;base64, — for example, data:;base64,<base64_encoded_client_secret>.

azureOAuthTokenEndpoint Body String Optional

The OAuth 2.0 token endpoint (v1.0), including the tenant ID, that the application uses to get an access token or a refresh token. Required if you are using the azureAuthenticationType AZURE_ACTIVE_DIRECTORY and Azure Storage as the storage provider for your Nessie source.

Example: https://login.microsoft.com/abc3e59a-7f07-4b2b-b894-b16d0EXAMPLE/oauth2/v2.0/token

googleProjectId Body String Optional

The ID for your Google Cloud Storage (GCS) project. You can find the ID in the Project info pane at the top-left of your screen on the GCS Home page. Required if you are using GCS as the storage provider for your Nessie source.

Example: project-id-1234

googleRootPath Body String Optional

The Google Cloud Storage (GCS) path prefix that Dremio should use as the write location for new Iceberg metadata and data. The root path includes the name of the GCS bucket, followed by the names of any folders. Required if you are using GCS as the storage provider for your Nessie source.

Example: gcs-data/prod/nessie_gcs

googleAuthenticationType Body String Optional

The method you want to use to authenticate to Google Cloud Storage (GCS): service account key (SERVICE_ACCOUNT_KEYS) or automatic/service account (AUTO). If you choose SERVICE_ACCOUNT_KEYS, you must also include the googleClientEmail, googleClientId, and googlePrivateKey parameters in your request. If you choose AUTO, and you are running Dremio on a Google Compute instance, Dremio uses the active service account for your instance and does not require any additional information to integrate with your data. Required if you are using GCS as the storage provider for your Nessie source.

Enum: AUTO, SERVICE_ACCOUNT_KEYS

Example: AUTO

googleClientEmail Body String Optional

The email address associated with the Google Cloud Storage (GCS) service account. Required if you are using googleAuthenticationType SERVICE_ACCOUNT_KEY and GCS as the storage provider for your Nessie source.

Example: gcs_prod@company.com

googleClientId Body String Optional

The client ID for your Google Cloud Storage (GCS) key pair. Required if you are using googleAuthenticationType SERVICE_ACCOUNT_KEY and GCS as the storage provider for your Nessie source.

Example: GA1.1.24470803.3400000000

googlePrivateKeyId Body String Optional

The key ID for your Google Cloud Storage key pair.

Example: ab7jqegoaz6ar4xplnf9vnxtoia86z0rnexample

googlePrivateKey Body String Optional

The key for your Google Cloud Storage (GCS) key pair in plaintext or as the reference for the Azure client secret in Azure Key Vault, AWS Secrets Manager, or HashiCorp Vault. Required if you are using googleAuthenticationType SERVICE_ACCOUNT_KEY and GCS as the storage provider for your Nessie source. To keep the key secure, Dremio returns the googlePrivateKey value as $DREMIO_EXISTING_VALUE$ in API responses.

Plaintext googlePrivateKey values must be prefixed with data:, — for example, data:,<plaintext_private_key>. Plaintext googlePrivateKey values that contain a colon (:) must be encoded in base64 format and prefixed with data:;base64, — for example, data:;base64,<base64_encoded_private_key>.

propertyList Body Array of Object Optional

Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.

Example: [{"name": "fs.s3a.proxy.host","value": "proxyHost.example.com"}]

secure Body Boolean Optional

If set to enable a secure connection with SSL encryption between the Amazon S3 bucket and Dremio, the value is true (default). Otherwise, the value is false.

Example: true

Parameters of Objects in the `propertyList` Array

name Body String Optional

Name of the connection property.

Example: fs.s3a.proxy.host

value Body String Optional

Value for the connection property.

Example: proxyHost.example.com

Oracle

Oracle Source config Object

{
  "config": {
    "hostname": "172.25.0.10",
    "port": "1521",
    "instance": "xe",
    "username": "exampleuser",
    "password": "$DREMIO_EXISTING_VALUE$",
    "authenticationType": "MASTER",
    "fetchSize": 200,
    "useSsl": false,
    "useTimezoneAsRegion": true,
    "includeSynonyms": false,
    "useLdap": false,
    "nativeEncryption": "ACCEPTED",
    "useKerberos": false,
    "maxIdleConns": 8,
    "idleTimeSec": 60,
    "mapDateToTimestamp": true,
    "queryTimeoutSec": 0,
    "propertyList": [
      {
        "name": "connect_timeout",
        "value": "0"
      }
    ]
  }
}

Oracle config Parameters

hostname Body String

Oracle hostname.

Example: 172.25.0.10

port Body String

Oracle port number. Default is 1521.

Example: 1521

instance Body String

Name of the Oracle database.

Example: xe

username Body String

Username for authentication. Required for authenticationType MASTER.

Example: exampleuser

password Body String

secretResourceUrl Body String

For secret-based authentication, the Amazon Resource Name (ARN) for the secret. To keep the secret secure, Dremio returns the secretResourceUrl value as $DREMIO_EXISTING_VALUE$ in API responses.

Example: arn:aws:secretsmanager:us-west-2:123456789012㊙️my-rds-secret-VNenFy

authenticationType Body String

Type of authentication to use to allow Dremio to connect to the Oracle source.

ANONYMOUS: No authentication is needed.
MASTER: Use credentials from a master database user or a secret resource URL.
KERBEROS: Authenticate with Kerberos.

Example: MASTER

fetchSize Body Integer Optional

Number of records to fetch at one time. Default is 200. If set to 0, Dremio automatically decides how many records to fetch.

Example: 200

useSsl Body Boolean Optional

If the Oracle source is configured to use TLS for connections, set to true. Otherwise, set to false (default).

Example: false

useTimezoneAsRegion Body Boolean Optional

To use the timezone to set the connection region, set to true (default). Otherwise, set to false.

Example: true

includeSynonyms Body Boolean Optional

To include Oracle synonyms as datasets in the source, set to true. Otherwise, set to false (default).

Example: false

nativeEncryption Body String Optional

If using Oracle native encryption for the connection, the encryption setting to use. Default is ACCEPTED.

Enum: REJECTED, ACCEPTED, REQUESTED, REQUIRED

Example: ACCEPTED

useKerberos Body Boolean Optional

To use Kerberos for authentication, set to true. Otherwise, set to false (default).

Example: false

maxIdleConns Body Integer Optional

Maximum number of connections that can be idle at any given time. Default is 8.

Example: 8

idleTimeSec Body Integer Optional

Maximum time that a connection can remain idle before Dremio terminates it, in seconds. Default is 60.

Example: 60

mapDateToTimestamp Body Boolean Optional

If the DATE column should display values in timestamp format, set to true (default). Otherwise, set to false.

Example: true

queryTimeoutSec Body Integer Optional

Maximum time to allow for query execution, in seconds. When the query timeout expires, the connection returns to an idle state. Default is 0.

Example: 0

propertyList Body Array of Object Optional

Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.

Example: [{"name": "connect_timeout","value": "0"}]

Parameters of Objects in the `propertyList` Array

name Body String Optional

Name of the connection property.

Example: connect_timeout

value Body String Optional

Value for the connection property.

Example: 0

Polaris Catalog

Polaris Catalog Source config Object

{
  "config": {
    "propertyList": [
      {
        "name": "fs.s3a.aws.credentials.provider",
        "value": "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"
      }
    ],
    "secretPropertyList": [
      {
        "name": "fs.s3a.access.key",
        "value": "$DREMIO_EXISTING_VALUE$"
      },
      {
        "name": "fs.s3a.secret.key",
        "value": "$DREMIO_EXISTING_VALUE$"
      }
    ],
    "enableAsync": true,
    "isCachingEnabled": true,
    "maxCacheSpacePct": 100,
    "restEndpointUri": "https://phy87032.snowflakecomputing.com/polaris/api/catalog",
    "allowedNamespaces": [
            "sales",
            "finance"
        ],
    "isRecursiveAllowedNamespaces": true,
    "polarisCatalogWarehouse": "polaris-catalog",
    "polarisClientId": "ImN1IMc4SvYjkB63X3aj0P8w7H86",
    "polarisClientSecret": "$DREMIO_EXISTING_VALUE$",
    "polarisScope": "PRINCIPAL_ROLE:ALL"
  }
}

Polaris Catalog config Parameters

propertyList Body Array of Object Optional

Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property. For a complete list of the catalog properties required for storage authentication, see Advanced Options on the Polaris Catalog page.

Example: [{"name": "fs.s3a.aws.credentials.provider","value": "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"}]

secretPropertyList Body Array of Object Optional

Additional connection properties for the source. The values are kept masked in the Credentials section of the Advanced Options page of the Source Settings dialog in the Dremio console. Each object includes the name of the property and the corresponding value to use and to keep secret. To keep the secret secure, Dremio returns the secretPropertyList value as $DREMIO_EXISTING_VALUE$ in API responses. For a complete list of the catalog credentials required for storage authentication, see Advanced Options on the Polaris Catalog page.

Example: [{"name": "fs.s3a.access.key","value": "$DREMIO_EXISTING_VALUE$"}]

enableAsync Body Boolean Optional

To enable asynchronous access for the source and allow cloud caching to support simultaneous actions such as adding and editing, set to true (default). Otherwise, set to false.

Example: true

isCachingEnabled Body Boolean Optional

To enable local caching, set to true (default). Otherwise, set to false.

Example: true

maxCacheSpacePct Body Integer Optional

Maximum percentage of the total available cache space to use on any single executor node. Used only when isCachingEnabled is set to true. Minimum is 1. Maximum is 100 (default).

Example: 100

restEndpointUri Body String

URI for the catalog service that Dremio should connect to.

Example: https://phy87032.snowflakecomputing.com/polaris/api/catalog

allowedNamespaces Body Array of String Optional

A list of the namespaces that users of Dremio are allowed to access. Tables are organized into namespaces, which can be at the top level or nested within one another. Namespace names cannot contain periods or spaces. Present the names in a comma-separated list.

Example: ["sales","finance”]

isRecursiveAllowedNamespaces Body Boolean Optional

If allowedNamespaces include their whole subtrees, set to true (default). Otherwise, set to false.

Example: true

polarisCatalogWarehouse Body String

The Polaris Catalog name.

Example: polaris-catalog

polarisClientId Body String

The client ID created during the configuration of a service connection in Snowflake's Polaris Catalog.

Example: ImN1IMc4SvYjkB63X3aj0P8w7H86

polarisClientSecret Body String

The client secret in plaintext or as the reference for the client secret in Azure Key Vault, AWS Secrets Manager, or HashiCorp Vault. To keep the client secret secure, Dremio returns the polarisClientSecret value as $DREMIO_EXISTING_VALUE$ in API responses.

Plaintext polarisClientSecret values must be prefixed with data:, — for example, data:,<plaintext_client_secret>. Plaintext polarisClientSecret values that contain a colon (:) must be encoded in base64 format and prefixed with data:;base64, — for example, data:;base64,<base64_encoded_client_secret>.

polarisScope Body String Optional

The OAuth 2.0 scope limit specifying the principal role that is allowed access to the Polaris Catalog.

Example: PRINCIPAL_ROLE:ALL

Parameters of Objects in the `propertyList` Array

name Body String Optional

Name of the connection property. For a complete list of the catalog properties required for storage authentication, see Advanced Options on the Polaris Catalog page.

Example: fs.s3a.aws.credentials.provider

value Body String Optional

Value for the connection property. For a complete list of the catalog properties required for storage authentication, see Advanced Options on the Polaris Catalog page.

Example: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider

Parameters of Objects in the `secretPropertyList` Array

name Body String Optional

Name of the connection property. For a complete list of the catalog credentials required for storage authentication, see Advanced Options on the Polaris Catalog page.

Example: fs.s3a.secret.key

value Body String Optional

Value for the connection property. The value is masked in the Dremio console. For a complete list of the catalog credentials required for storage authentication, see Advanced Options on the Polaris Catalog page.

Example: l1hDG6w6E1w1LgTrRwDwxNaiWV5wn18210536VhD

PostgreSQL

PostgreSQL Source config Object

{
  "config": {
    "hostname": "172.25.0.10",
    "port": "5432",
    "databaseName": "postgres",
    "username": "exampleuser",
    "password": "$DREMIO_EXISTING_VALUE$",
    "authenticationType": "MASTER",
    "fetchSize": 200,
    "useSsl": false,
    "encryptionValidationMode": "CERTIFICATE_AND_HOSTNAME_VALIDATION",
    "maxIdleConns": 8,
    "idleTimeSec": 60,
    "queryTimeoutSec": 0,
    "propertyList": [
      {
        "name": "connect_timeout",
        "value": "10"
      }
    ]
  }
}

PostgreSQLconfig Parameters

hostname Body String

PostgreSQL hostname.

Example: 172.25.0.10

port Body String

PostgreSQL port number. Default is 5432.

Example: 5432

databaseName Body String

Default PostgreSQL database for Dremio to use.

Example: postgres

username Body String

Username for authentication. Required for authenticationType MASTER.

Example: exampleuser

password Body String

secretResourceUrl Body String

For secret-based authentication, the Amazon Resource Name (ARN) for the secret. To keep the secret secure, Dremio returns the secretResourceUrl value as $DREMIO_EXISTING_VALUE$ in API responses.

Example: arn:aws:secretsmanager:us-west-2:123456789012㊙️my-rds-secret-VNenFy

authenticationType Body String

Type of authentication to use to allow Dremio to connect to the PostgreSQL source.

ANONYMOUS: No authentication is needed.
MASTER: Use credentials from a master database user or a secret resource URL.

Example: MASTER

fetchSize Body Integer Optional

Number of records to fetch at one time. Default is 200. If set to 0, Dremio automatically decides how many records to fetch.

Example: 200

useSsl Body Boolean Optional

If the PostgreSQL source is configured to use TLS for connections, set to true. Otherwise, set to false.

Example: false

encryptionValidationMode Body String Optional

Method to use to validate data encryption for the source. The available settings map to PostgreSQL sslmode terminology as follows:

CERTIFICATE_AND_HOSTNAME_VALIDATION: verify-full
CERTIFICATE_ONLY_VALIDATION: verify-ca
NO_VALIDATION: require

Example: CERTIFICATE_AND_HOSTNAME_VALIDATION

maxIdleConns Body Integer Optional

Maximum number of connections that can be idle at any given time. Default is 8.

Example: 8

idleTimeSec Body Integer Optional

Maximum time that a connection can remain idle before Dremio terminates it, in seconds. Default is 60.

Example: 60

queryTimeoutSec Body Integer Optional

Maximum time to allow for query execution, in seconds. When the query timeout expires, the connection returns to an idle state. Default is 0.

Example: 0

propertyList Body Array of Object Optional

Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.

Example: [{"name": "connect_timeout","value": "10&application_name=myapp"}]

Parameters of Objects in the `propertyList` Array

name Body String Optional

Name of the connection property.

Example: connect_timeout

value Body String Optional

Value for the connection property.

Example: 10

Snowflake

Snowflake Source config Object

{
  "config": {
    "hostname": "https://locator_id.snowflakecomputing.com",
    "port": "443",
    "database": "sample_data",
    "warehouse": "compute_wh",
    "username": "dremio",
    "maxIdleConns": 8,
    "idleTimeSec": 60,
    "queryTimeoutSec": 0,
    "fetchSize": 2000,
    "authMode": "KEY_PAIR",
    "privateKey": "data:,fa2dlf8MaczJj+szv5KSCvQzs5MGhNtiMggV5qcw8vkgRshfZw76FKBPrZu4=ap35Eg",
    "propertyList": [
      {
        "name": "loginTimeout",
        "value": "60"
      }
    ]
  }
}

Snowflake config Parameters

hostname Body String

The Snowflake host name.

Example: https://locator_id.snowflakecomputing.com

port Body String

The Snowflake port number. The commonly used default port number for Snowflake is 443.

Example: 443

database Body String Optional

Default Snowflake database for Dremio to use.

Example: sample_data

warehouse Body String Optional

Snowflake virtual warehouse to use to provide resources for executing DML statements and queries.

Example: compute_wh

username Body String

Username for the Snowflake account.

Example: dremio

password Body String

Password for the Snowflake account. Required if authMode is LOGIN_PASSWORD.

You can provide the password in plaintext or as the reference for the password in Azure Key Vault, AWS Secrets Manager, or HashiCorp Vault. To keep the password secure, Dremio returns the password value as $DREMIO_EXISTING_VALUE$ in API responses.

maxIdleConns Body Integer Optional

Maximum number of connections that can be idle at any given time. Default is 8.

Example: 8

idleTimeSec Body Integer Optional

Maximum time that a connection can remain idle before Dremio terminates it, in seconds. Default is 60.

Example: 60

queryTimeoutSec Body Integer Optional

Maximum time to allow for query execution, in seconds. When the query timeout expires, the connection returns to an idle state. Default is 0.

Example: 0

fetchSize Body Integer Optional

The number of records to fetch per request. Set to 0 to enable Dremio to automatically set the size. Default is 2000.

Example: 2000

authMode Body String Optional

Type of authentication for Dremio to use to connect to the source. If you do not specify an authMode, the default is LOGIN_PASSWORD.

Enum: LOGIN_PASSWORD, KEY_PAIR

Example: KEY_PAIR

privateKey Body String

The generated Snowflake private key in Privacy Enhanced Mail (PEM) format. Required if authMode is KEY_PAIR. To keep the privateKey secure, Dremio returns the privateKey value as $DREMIO_EXISTING_VALUE$ in API responses.

You can provide the privateKey in plaintext or as the reference for the privateKey in Azure Key Vault, AWS Secrets Manager, or HashiCorp Vault.

Plaintext privateKey values must be prefixed with data:, — for example, data:,<plaintext_privateKey>. Plaintext privateKey values that contain a colon (:) must be encoded in base64 format and prefixed with data:;base64, — for example, data:;base64,<base64_encoded_privateKey>.

Example: data:,fa2dlf8MaczJj+szv5KSCvQzs5MGhNtiMggV5qcw8vkgRshfZw76FKBPrZu4=ap35Eg

privateKeyPassphrase Body String Optional

The passphrase for an encrypted private key. Required if authMode is KEY_PAIR and the privateKey value is an encrypted private key.

You can provide the privateKeyPassphrase in plaintext or as the reference for the privateKeyPassphrase in Azure Key Vault, AWS Secrets Manager, or HashiCorp Vault.

Plaintext privateKeyPassphrase values must be prefixed with data:, — for example, data:,<plaintext_privateKeyPassphrase>. Plaintext password values that contain a colon (:) must be encoded in base64 format and prefixed with data:;base64, — for example, data:;base64,<base64_encoded_privateKeyPassphrase>.

Example: data:,!3RoX$=w72G=e%G64p#ozm

propertyList Body Array of Object Optional

Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.

Example: [{"name": "loginTimeout","value": "60"}]

Parameters of Objects in the `propertyList` Array

name Body String Optional

Name of the connection property.

Example: loginTimeout

value Body String Optional

Value for the connection property.

Example: 60

Teradata

Teradata Source config Object

{
  "config": {
    "hostname": "teradata.drem.io",
    "port": "1025",
    "database": "dremio_source",
    "username": "dremio",
    "password": "$DREMIO_EXISTING_VALUE$",
    "fetchSize": 200,
    "showOnlyConnectionDatabase": false,
    "useSsl": false,
    "maxIdleConns": 8,
    "idleTimeSec": 60,
    "queryTimeoutSec": 0,
    "propertyList": [
      {
        "name": "maxPoolSize",
        "value": "10"
      }
    ]
  }
}

Teradata config Parameters

hostname Body String

Teradata hostname.

Example: teradata.drem.io

port Body String

Teradata port number. Default is 5432.

Example: 1025

database Body String Optional

Service name of the Teradata database.

Example: dremio_source

username Body String

Username for authentication.

Example: dremio

password Body String

fetchSize Body Integer Optional

Number of records to fetch at one time. Default is 200. If set to 0, Dremio automatically decides how many records to fetch.

Example: 200

showOnlyConnectionDatabase Body Boolean Optional

To hide all databases that the credentialed user can access except the specified connection database, set to true. Otherwise, set to false (default).

Example: false

useSsl Body Boolean Optional

If the Teradata source is configured to use TLS for connections, set to true. Otherwise, set to false (default).

Example: false

maxIdleConns Body Integer Optional

Maximum number of connections that can be idle at any given time. Default is 8.

Example: 8

idleTimeSec Body Integer Optional

Maximum time that a connection can remain idle before Dremio terminates it, in seconds. Default is 60.

Example: 60

queryTimeoutSec Body Integer Optional

Maximum time to allow for query execution, in seconds. When the query timeout expires, the connection returns to an idle state. Default is 0.

Example: 0

propertyList Body Array of Object Optional

Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.

Example: [{"name": "maxPoolSize","value": "10"}]

Parameters of Objects in the `propertyList` Array

name Body String Optional

Name of the connection property.

Example: maxPoolSize

value Body String Optional

Value for the connection property.

Example: 10

Unity Catalog

Unity Catalog config Object

{
  "config": {
    "propertyList": [
      {
        "name": "fs.s3a.aws.credentials.provider",
        "value": "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"
      }
    ],
    "secretPropertyList": [
      {
        "name": "fs.s3a.access.key",
        "value": "$DREMIO_EXISTING_VALUE$"
      },
      {
        "name": "fs.s3a.secret.key",
        "value": "$DREMIO_EXISTING_VALUE$"
      }
    ],
    "enableAsync": true,
    "isCachingEnabled": true,
    "maxCacheSpacePct": 100,
    "restEndpointUri": "https://dbc-3i6c5lto-k3v3.cloud.databricks.com/api/2.1/unity-catalog/iceberg",
    "allowedNamespaces": [
           "sales",
           "finance"
       ],
    "isRecursiveAllowedNamespaces": true,
    "unityAuthToken": "$DREMIO_EXISTING_VALUE$",
    "unityCatalog": "unity_catalog"
  }
}

Unity Catalog config Parameters

propertyList Body Array of Object Optional

Example: [{"name": "fs.s3a.aws.credentials.provider","org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"}]

secretPropertyList Body Array of Object Optional

Example: [{"name": "fs.azure.account.key","value": "70SPup32UsIZaA6c2n6bf3rQONTD6Zn6OqvcUhBy"}]

enableAsync Body Boolean Optional

To enable asynchronous access for the source and allow cloud caching to support simultaneous actions such as adding and editing, set to true (default). Otherwise, set to false.

Example: true

isCachingEnabled Body Boolean Optional

To enable local caching, set to true (default). Otherwise, set to false.

Example: true

maxCacheSpacePct Body Integer Optional

Maximum percentage of the total available cache space to use on any single executor node. Used only when isCachingEnabled is set to true. Minimum is 1. Maximum is 100 (default).

Example: 100

restEndpointUri Body String

URI for the catalog service that Dremio should connect to.

Example: https://dbc-3i6c5lto-k3v3.cloud.databricks.com/api/2.1/unity-catalog/iceberg

allowedNamespaces Body Array of String Optional

Example: ["sales","finance”]

isRecursiveAllowedNamespaces Body Boolean Optional

If allowedNamespaces include their whole subtrees, set to true (default). Otherwise, set to false.

Example: true

unityAuthToken Body String

unityCatalog Body String

Name of the Unity Catalog.

Example: unity_catalog

Parameters of Objects in the `propertyList` Array

name Body String Optional

Name of the connection property. For a complete list of the catalog properties required for storage authentication, see Advanced Options on the Unity Catalog page.

Example: fs.s3a.aws.credentials.provider

value Body String Optional

Value for the connection property. For a complete list of the catalog properties required for storage authentication, see Advanced Options on the Unity Catalog page.

Example: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider

Parameters of Objects in the `secretPropertyList` Array

name Body String Optional

Name of the connection property. For a complete list of the catalog credentials required for storage authentication, see Advanced Options on the Unity Catalog page.

Example: fs.s3a.access.key

value Body String Optional

Example: e0820ta9vgZlhOB9P9VABCbp7bsd8vP4Miqxisk5BUq9ZrqbufF1QvA5b99m0848ulmTbAUwG2wNZUY1vr8yoiKe

Vertica

Vertica Source config Object

{
  "config": {
    "database": "vertica",
    "hostname": "172.25.0.10",
    "username": "exampleuser",
    "password": "$DREMIO_EXISTING_VALUE$",
    "port": "5433",
    "fetchSize": 200,
    "maxIdleConns": 8,
    "idleTimeSec": 60,
    "queryTimeoutSec": 0,
    "propertyList": [
      {
        "name": "LoginTimeout",
        "value": "30"
      }
    ]
  }
}

Vertica config Parameters

database Body String

Default Vertica database for Dremio to use.

Example: vertica

hostname Body String

Vertica hostname.

Example: 172.25.0.10

username Body String

Username for authentication.

Example: exampleuser

password Body String

port Body String

Vertica port number. Default is 5433.

Example: 5433

fetchSize Body Integer Optional

Number of records to fetch at one time. Default is 200. If set to 0, Dremio automatically decides how many records to fetch.

Example: 200

maxIdleConns Body Integer Optional

Maximum number of connections that can be idle at any given time. Default is 8.

Example: 8

idleTimeSec Body Integer Optional

Maximum time that a connection can remain idle before Dremio terminates it, in seconds. Default is 60.

Example: 60

queryTimeoutSec Body Integer Optional

Maximum time to allow for query execution, in seconds. When the query timeout expires, the connection returns to an idle state. Default is 0.

Example: 0

propertyList Body Array of Object Optional

Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.

Example: [{"name": "LoginTimeout","value": "30&application_name=myapp"}]

Parameters of Objects in the `propertyList` Array

name Body String Optional

Name of the connection property.

Example: LoginTimeout

value Body String Optional

Value for the connection property.

Example: 30

Amazon OpenSearch Service​

Amazon OpenSearch Service config Parameters​

Amazon Redshift​

Amazon Redshift config Parameters​

Parameters of Objects in the propertyList Array​

Amazon S3​

Amazon S3 config Parameters​

Parameters of Objects in the propertyList Array​

AWS Glue Data Catalog​

Amazon Glue Data Catalog config Parameters​

Parameters of Objects in the propertyList Array​

Azure Storage​

Azure Storage config Parameters​

Parameters of Objects in the propertyList Array​

Dremio-to-Dremio Connector​

Dremio-to-Dremio Connector config Parameters​

Parameters of Objects in the propertyList Array​

Elasticsearch​

Elasticsearch config Parameters​

Parameters of Objects in the hostList Array​

Google Cloud Storage​

Google Cloud Storage config Parameters​

Parameters of Objects in the propertyList Array​

Hadoop Distributed File System (HDFS)​

HDFS config Parameters​

Parameters of Objects in the propertyList Array​

Hive 2.x​

Hive 2.x config Parameters​

Parameters of Objects in the propertyList Array​

Parameters of Objects in the secretPropertyList Array​

Hive 3.x​

Hive 3.x config Parameters​

Parameters of Objects in the propertyList Array​

Parameters of Objects in the secretPropertyList Array​

IBM Db2​

IBM Db2 config Parameters​

Parameters of Objects in the propertyList Array​

MapR File System (MapR FS)​

MapR FS config Parameters​

Microsoft Azure Data Explorer​

Microsoft Azure Date Explorer config Parameters​

Microsoft Azure Synapse Analytics​

Microsoft Azure Synapse Analytics config Parameters​

Parameters of Objects in the propertyList Array​

Microsoft SQL Server​

Microsoft SQL Server config Parameters​

Parameters of Objects in the propertyList Array​

MongoDB​

MongoDB config Parameters​

Parameters of Objects in the hostList Array​

Parameters of Objects in the propertyList Array​

MySQL​

mySQL config Parameters​

Parameters of Objects in the propertyList Array​

Network-attached Storage (NAS)​

NAS config Parameters​

Nessie​

Nessie config Parameters​

Parameters of Objects in the propertyList Array​

Oracle​

Oracle config Parameters​

Parameters of Objects in the propertyList Array​

Polaris Catalog​

Polaris Catalog config Parameters​

Parameters of Objects in the propertyList Array​

Parameters of Objects in the secretPropertyList Array​

PostgreSQL​

PostgreSQLconfig Parameters​

Parameters of Objects in the propertyList Array​

Snowflake​

Snowflake config Parameters​

Parameters of Objects in the propertyList Array​

Teradata​

Teradata config Parameters​

Parameters of Objects in the propertyList Array​

Unity Catalog​

Unity Catalog config Parameters​

Parameters of Objects in the propertyList Array​

Parameters of Objects in the secretPropertyList Array​

Vertica​

Amazon OpenSearch Service

Amazon OpenSearch Service config Parameters

Amazon Redshift

Amazon Redshift config Parameters

Parameters of Objects in the `propertyList` Array

Amazon S3

Amazon S3 config Parameters

Parameters of Objects in the `propertyList` Array

AWS Glue Data Catalog

Amazon Glue Data Catalog config Parameters

Parameters of Objects in the `propertyList` Array

Azure Storage

Azure Storage config Parameters

Parameters of Objects in the `propertyList` Array

Dremio-to-Dremio Connector

Dremio-to-Dremio Connector config Parameters

Parameters of Objects in the `propertyList` Array

Elasticsearch

Elasticsearch config Parameters

Parameters of Objects in the `hostList` Array

Google Cloud Storage

Google Cloud Storage config Parameters

Parameters of Objects in the `propertyList` Array

Hadoop Distributed File System (HDFS)

HDFS config Parameters

Parameters of Objects in the `propertyList` Array

Hive 2.x

Hive 2.x config Parameters

Parameters of Objects in the `propertyList` Array

Parameters of Objects in the `secretPropertyList` Array

Hive 3.x

Hive 3.x config Parameters

Parameters of Objects in the `propertyList` Array

Parameters of Objects in the `secretPropertyList` Array

IBM Db2

IBM Db2 config Parameters

Parameters of Objects in the `propertyList` Array

MapR File System (MapR FS)

MapR FS config Parameters

Microsoft Azure Data Explorer

Microsoft Azure Date Explorer config Parameters

Microsoft Azure Synapse Analytics

Microsoft Azure Synapse Analytics config Parameters

Parameters of Objects in the `propertyList` Array

Microsoft SQL Server

Microsoft SQL Server config Parameters

Parameters of Objects in the `propertyList` Array

MongoDB

MongoDB config Parameters

Parameters of Objects in the `hostList` Array

Parameters of Objects in the `propertyList` Array

MySQL

mySQL config Parameters

Parameters of Objects in the `propertyList` Array

Network-attached Storage (NAS)

NAS config Parameters

Nessie

Nessie config Parameters

Parameters of Objects in the `propertyList` Array

Oracle

Oracle config Parameters

Parameters of Objects in the `propertyList` Array

Polaris Catalog

Polaris Catalog config Parameters

Parameters of Objects in the `propertyList` Array

Parameters of Objects in the `secretPropertyList` Array

PostgreSQL

PostgreSQLconfig Parameters

Parameters of Objects in the `propertyList` Array

Snowflake

Snowflake config Parameters

Parameters of Objects in the `propertyList` Array

Teradata

Teradata config Parameters

Parameters of Objects in the `propertyList` Array

Unity Catalog

Unity Catalog config Parameters

Parameters of Objects in the `propertyList` Array

Parameters of Objects in the `secretPropertyList` Array

Vertica