Source Configuration

Dremio supports a variety of source types, and the available parameters of the config object differ among source types. This page provides examples of the config object for each supported source type and descriptions of the config object's parameters that are available for each source type.

The source object contains the same parameters for all source types except for the parameters of the config object. See the Source page for information about the parameters all source types have in common.

Amazon Redshift

Dremio source type: REDSHIFT
See Connect to Amazon Redshift for additional information.

Config attributes for Amazon Redshift

"config": {
  "connectionString": "jdbc:redshift://redshift.example.com:5439/database",
  "authenticationType": "MASTER",
  "username": "myUsername",
  "password": "myPassword",
  "secretResourceUrl": "arn:aws:secretsmanager:us-west-2:123456789012:secret:my-rds-secret-VNenFy",
  "awsProfile": "profileName",
  "dbUser": "userName",
  "fetchSize": 200,
  "maxIdleConns": 8,
  "idleTimeSec": 60,
  "queryTimeoutSec": 0,
  "propertyList": [
    {
      "name": "ApplicationName",
      "value": "MarketingDashboard"
    }
  ]
}

Parameters

connectionString Body String

The JDBC connection string. Read Dremio JDBC Driver (Legacy) for more information.

Example: jdbc:redshift://redshift.example.com:5439/database

authenticationType Body String

Specifies the type of authentication to enable Dremio to connect to the source:

ANONYMOUS: Use when no authentication is needed.
MASTER: Use when either credentials from a master database user or a secret resource URL is required.
AWS_PROFILE: Use for authentication to an AWS source.

username Body String

The Amazon Redshift username.

Example: myUsername

password Body String

The Amazon Redshift password.

Example: myPassword

secretResourceUrl Body String

For secret-based authentication, the Amazon Resource Name (ARN) for the secret. To keep the secret secure, Dremio returns the secretResourceUrl value as $DREMIO_EXISTING_VALUE$ in API responses.

Example: arn:aws:secretsmanager:us-west-2:123456789012:secret:my-rds-secret-VNenFy

awsProfile Body String

The AWS profile name. If this is left blank, the default profile will be used.

Example: profileName

dbUser Body String

The name of the Amazon Redshift DbUser that will be used for authentication. If this is left blank, the default username for your AWS IAM role will be used.

Example: userName

fetchSize Body Integer

The number of records to fetch per request. Set to 0 to enable Dremio to automatically set the size.

Example: 200

maxIdleConns Body Integer

Sets the maximum number of idle connections to keep.

Example: 8

idleTimeSec Body Integer

Sets the idle time, in seconds, before a connection is evaluated for closure.

Example: 60

queryTimeoutSec Body Integer

Sets the timeout, in seconds, for query execution before it is canceled. If you do not want a timeout, enter 0.

Example: 0

propertyList Body Array of Object Optional

An array of name/value pairs defining optional connection properties. Use a comma-separated list to specify multiple name/value pairs.

Example:

{
  "name": "ApplicationName",
  "value": "MarketingDashboard"
}

Authentication Examples

No authentication required

{
  "authenticationType": "ANONYMOUS"
}

Master credentials

{
  "authenticationType": "MASTER",
  "username": "myUsername",
  "password": "myPassword"
}

Secret resource URL

{
  "authenticationType": "MASTER",
  "username": "myUsername",
  "secretResourceUrl": "arn:aws:secretsmanager:us-west-2:123456789012:secret:my-rds-secret-VNenFy"
}

AWS Profile

{
  "authenticationType": "AWS_PROFILE",
  "awsProfile": "profileName",
  "dbUser": "userName"
}

Amazon S3

Dremio source type: S3
See Amazon S3 for additional information.

Config attributes for Amazon S3

"config": {
  "credentialType": "ACCESS_KEY",
  "assumedRoleARN": "arn:aws:iam::123456789012:root",
  "awsAccessKey": "AKIAIOSFODNN7EXAMPLE",
  "awsAccessSecret": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
  "externalBucketList": [ "bucket-name" ],
  "secure": false,
  "enableAsync": true,
  "compatibilityMode": false,
  "requesterPays": false,
  "enableFileStatusCheck": true,
  "isPartitionInferenceEnabled": false,
  "rootPath": "/",
  "kmsKeyARN": "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab",
  "defaultCtasFormat": "ICEBERG",
  "propertyList": [
    {
      "name": "fs.s3a.proxy.host",
      "value": "proxyHost.example.com"
    }
  ],
  "whitelistedBuckets": [
    "sales"
  ],
  "isCachingEnabled": true,
  "maxCacheSpacePct": 100
}

Parameters

credentialType Body String

Specifies the type of credential to enable Dremio to connect to the source:

ACCESS_KEY: Use an ACCESS_KEY to directly authenticate to the source bucket, or to assume a role for bucket authentication.
PROJECT_DATA_CRED: Use the project data credentials to directly authenticate to the source bucket, or to assume a role for bucket authentication.

assumedRoleARN Body String Optional

The ARN of the IAM role to assume.

Example: arn:aws:iam::123456789012:root

awsAccessKey Body String

The AWS access key.

Example: AKIAIOSFODNN7EXAMPLE

awsAccessSecret Body String

The AWS access secret.

Example: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

externalBucketList Body Array of String

Connects Dremio to one or more Amazon S3 buckets. See Accessing a bucket using S3 for more information.

Example:

[
  "bucket-name"
]

secure Body Boolean

Option to enable a secure connection via SSL encryption between the S3 buckets and Dremio. Set to true to enable SSL encryption.

enableAsync Body Boolean

Option to enable asynchronous access, which allows cloud caching for the source to support simultaneous actions such as adding and editing. Set to true to enable asynchronous access. See Amazon S3 > Advanced Options for more information.

compatibilityMode Body Boolean

Option to enable compatibility mode for an Amazon S3 source, which allows you to use S3-compatible storage. Set to true to enable compatibility mode.

requesterPays Body Boolean

Option to apply S3 requests to the requester. Set to true to enable this option.

enableFileStatusCheck Body Boolean

Option to enable file status check. Set to true to enable file status check.

isPartitionInferenceEnabled Body Boolean

Option to enable partition column inference. Set to true to enable partition column inference.

rootPath Body String

The root path of the S3 bucket.

Example: /

kmsKeyARN Body String

Enables you to add a server-side encryption key ARN that was created in AWS Key Management Service (KMS).

Example: arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab

defaultCtasFormat Body String

Enables you to choose the default format for tables you create in Dremio, either ICEBERG or PARQUET.

propertyList Body Array of Object Optional

An array of name/value pairs defining optional connection properties. Use a comma-separated list to specify multiple name/value pairs.

Example:

{
  "name": "fs.s3a.proxy.host",
  "value": "proxyHost.example.com"
}

whitelistedBuckets Body Array of String

List of approved S3 buckets to include in the source. Buckets entered must be valid. Misspelled or nonexistent buckets will not appear in the resulting source. Omit to include all buckets that are available in the source. If you omit the whitelistedBuckets array in a PUT request, Dremio updates the source to include all available buckets. To keep existing approved buckets while making other updates, duplicate the existing whitelistedBuckets array in the PUT request.

Example:

[
  "archive.dremio.com",
  "logs_east-1",
  "logs_west-1"
]

isCachingEnabled Body Boolean

Option to enable local caching. When local caching is possible, set this parameter to true to enable it. See Amazon S3 > Advanced Options for more information.

maxCacheSpacePct Body Integer

Sets the maximum percent of total available cache space to use when possible. The value, in percent, must be between 1 and 100.

Example: 100

Authentication Examples

Project data credentials

{
  "credentialType": "PROJECT_DATA_CRED"
}

Data source credentials (IAM role)

{
  "credentialType": "PROJECT_DATA_CRED",
  "assumedRoleARN": "arn:aws:iam::123456789012:root"
}

Data source credentials (access key)

{
  "credentialType": "ACCESS_KEY",
  "assumedRoleARN": "arn:aws:iam::123456789012:root",
  "awsAccessKey": "AKIAIOSFODNN7EXAMPLE",
  "awsAccessSecret": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
}

AWS Glue Data Catalog

Dremio source type: AWSGLUE
See Connect to an AWS Glue Data Catalog for additional information.

Config attributes for AWS Glue Data Catalog

"config": {
  "credentialType": "ACCESS_KEY",
  "assumedRoleARN": "arn:aws:iam::123456789012:root",
  "awsAccessKey": "AKIAIOSFODNN7EXAMPLE",
  "awsAccessSecret": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
  "regionNameSelection": "US_EAST_1",
  "secure": false,
  "enableAsync": true,
  "lakeFormationEnableAccessPermissions": false,
  "propertyList": [
   {
      "name": "hive.metastore.warehouse.dir",
      "value": "s3://bucket-name/folder"
    }
  ],
  "allowedDatabases": ["sales", "finance"],
  "isCachingEnabled": true,
  "maxCacheSpacePct": 100
}

Parameters

credentialType Body String

Specifies the type of credential to enable Dremio to connect to the source:

ACCESS_KEY: Use an ACCESS_KEY to directly authenticate to the source bucket, or to assume a role for bucket authentication.
PROJECT_DATA_CRED: Use the project data credentials to directly authenticate to the source bucket, or to assume a role for bucket authentication.

assumedRoleARN Body String Optional

The ARN of the IAM role to assume.

Example: arn:aws:iam::123456789012:root

awsAccessKey Body String

The AWS access key.

Example: AKIAIOSFODNN7EXAMPLE

awsAccessSecret Body String

The AWS access secret.

Example: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

regionNameSelection Body String

Sets the AWS Region where the AWS Glue Data Catalog is located.

Valid Values: AF_SOUTH_1, AP_EAST_1, AP_NORTHEAST_1, AP_NORTHEAST_2, AP_NORTHEAST_3, AP_SOUTHEAST_1, AP_SOUTHEAST_2, AP_SOUTHEAST_3, AP_SOUTHEAST_4, AP_SOUTH_1, AP_SOUTH_2, CA_CENTRAL_1, CN_NORTHWEST_1, CN_NORTH_1, EU_CENTRAL_1, EU_CENTRAL_2, EU_NORTH_1, EU_SOUTH_1, EU_SOUTH_2, EU_WEST_1, EU_WEST_2, EU_WEST_3, IL_CENTRAL_1, ME_CENTRAL_1, ME_SOUTH_1, SA_EAST_1, US_EAST_1, US_EAST_2, US_GOV_EAST_1, US_GOV_WEST_1, US_WEST_1, US_WEST_2

Example: US_EAST_1

secure Body Boolean

Option to enable a secure connection via SSL encryption between the AWS Glue Data Catalog and Dremio. Set to true to enable SSL encryption.

enableAsync Body Boolean

lakeFormationEnableAccessPermissions Body Boolean

Set to true to enforce Lake Formation access permissions on datasets so that Dremio can confirm whether users have the required permissions to perform queries. Otherwise, set to false (default).

propertyList Body Array of Object Optional

An array of name/value pairs defining optional connection properties. Use a comma-separated list to specify multiple name/value pairs.

Example:

[
  {
    "name": "hive.metastore.warehouse.dir",
    "value": "s3://bucket-name/folder"
  }
]

allowedDatabases Body Array of String

A list of the databases that users of Dremio are allowed to access. Present the names in a comma-separated list. Databases entered must be valid. Misspelled or nonexistent databases will not appear in the resulting source.

If one or more databases are added or removed from the list, the change takes effect immediately.

Example:

[
  "sales",
  "finance"
]

isCachingEnabled Body Boolean

Option to enable local caching. When local caching is possible, set this parameter to true to enable it. See AWS Glue Data Catalog > Advanced Options for more information.

maxCacheSpacePct Body Integer

Sets the maximum percent of total available cache space to use when possible. The value, in percent, must be between 1 and 100.

Example: 100

Authentication Examples

Project data credentials

{
  "credentialType": "PROJECT_DATA_CRED"
}

Data source credentials (IAM role)

{
  "credentialType": "PROJECT_DATA_CRED",
  "assumedRoleARN": "arn:aws:iam::123456789012:root"
}

Data source credentials (access key)

{
  "credentialType": "ACCESS_KEY",
  "assumedRoleARN": "arn:aws:iam::123456789012:root",
  "awsAccessKey": "AKIAIOSFODNN7EXAMPLE",
  "awsAccessSecret": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
}

Open Catalog

Dremio source type: DCS_DREMIO_CATALOG_V1
See Open Catalog for additional information.

Config attributes for Dremio's Open Catalog

{
  "config": {
    "enableAsync": true,
    "isCachingEnabled": true,
    "maxCacheSpacePct": 100,
    "isRecursiveAllowedNamespaces": true,
    "isUsingVendedCredentials": true
  }
}

Parameters

enableAsync Body Boolean

Option to enable asynchronous access, which allows cloud caching for the source to support simultaneous actions such as adding and editing. Set to true to enable asynchronous access.

isCachingEnabled Body Boolean

Option to enable local caching. When local caching is possible, set this parameter to true to enable it.

maxCacheSpacePct Body Integer

Sets the maximum percent of total available cache space to use when possible. The value, in percent, must be between 1 and 100.

Example: 100

isRecursiveAllowedNamespaces Body Boolean Optional

If allowedNamespaces include their whole subtrees, set to true (default). Otherwise, set to false.

isUsingVendedCredentials Body Boolean

If set to true, you do not need to provide credentials via the secretPropertyList object in your request, as the catalog will return temporary credentials for the underlying storage location. If set to false, credentials must be provided for the underlying storage location of the catalog to which you are connecting.

Google Cloud Lakehouse Catalog

Dremio source type: RESTCATALOG
See Connect to a Google Cloud Lakehouse Catalog for additional information.

Config attributes for a Google Cloud Lakehouse Catalog

{
  "config": {
    "propertyList": [
      {
        "name": "warehouse",
        "value": "your-warehouse"
      },
      {
        "name": "rest.auth.type",
        "value": "org.apache.iceberg.gcp.auth.GoogleAuthManager"
      },
      {
        "name": "header.x-goog-user-project",
        "value": "googleprojectname"
      }
    ],
    "secretPropertyList": [
      {
        "name": "gcp.auth.credentials-json",
        "value": "<your_ADC_JSON_here>"
      }
    ],
    "enableAsync": true,
    "isCachingEnabled": true,
    "maxCacheSpacePct": 100,
    "restEndpointUri": "https://biglake.googleapis.com/iceberg/v1/restcatalog",
    "allowedNamespaces": [
      "your-namespace"
    ],
    "isUsingVendedCredentials": true,
    "isRecursiveAllowedNamespaces": false
  }
}

Parameters

propertyList Body Array of Object Optional

Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property. The properties captured above are an example of connecting to a Cloud Lakehouse catalog with vended credentials.

For a complete list of the catalog properties required for storage authentication, see Advanced Options on the Google Cloud Lakehouse Catalog page.

Example:

[
  {
    "name": "rest.auth.type",
    "value": "org.apache.iceberg.gcp.auth.GoogleAuthManager"
  }
]

secretPropertyList Body Array of Object Optional

Additional connection properties for the source. The values are kept masked in the Credentials section of the Advanced Options page of the Source Settings dialog in the Dremio console. Each object includes the name of the property and the corresponding value to use and to keep secret. To keep the secret secure, Dremio returns the secretPropertyList value as $DREMIO_EXISTING_VALUE$ in API responses.

Example:

[
  {
    "name": "gcp.auth.credentials-json",
    "value": "$DREMIO_EXISTING_VALUE$"
  }
]

enableAsync Body Boolean Optional

To enable asynchronous access for the source and allow cloud caching to support simultaneous actions such as adding and editing, set to true (default). Otherwise, set to false.

isCachingEnabled Body Boolean Optional

To enable local caching, set to true (default). Otherwise, set to false.

maxCacheSpacePct Body Integer Optional

Maximum percentage of the total available cache space to use on any single executor node. Used only when isCachingEnabled is set to true. Minimum is 1. Maximum is 100 (default).

Example: 100

restEndpointUri Body String

URI for the catalog service that Dremio should connect to.

Example: https://biglake.googleapis.com/iceberg/v1/restcatalog

isUsingVendedCredentials Body Boolean

allowedNamespaces Body Array of String Optional

A list of the namespaces that Dremio users are allowed to access. Tables are organized into namespaces, which can be at the top level or nested within one another. Namespace names cannot contain periods or spaces. Present the names in a comma-separated list.

Example:

[
  "your-namespace"
]

isRecursiveAllowedNamespaces Body Boolean Optional

If allowedNamespaces include their whole subtrees, set to true (default). Otherwise, set to false.

Parameters of Objects in the `propertyList` Array

name Body String Optional

Name of the connection property.

Example: header.x-goog-user-project

value Body String Optional

Value for the connection property.

Example: googleprojectname

Parameters of Objects in the `secretPropertyList` Array

name Body String Optional

Name of the connection property.

Example: gcp.auth.credentials-json

value Body String Optional

Value for the connection property. The value is masked in the Dremio console. Note that you will need to escape any " charecters in your ADC json with \.

Example: <your_ADC_JSON_here>

Google BigQuery

Dremio source type: BIGQUERY
See Connect to Google BigQuery for additional information.

Config attributes for Google BigQuery

{
  "config": {
    "hostname": "https://www.googleapis.com/bigquery/v2",
    "port": "443",
    "projectId": "{project_id}",
    "authMode": "SERVICE_ACCOUNT_KEYS",
    "clientEmail": "service-account@your_project.iam.gserviceaccount.com",
    "privateKey": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----",
    "fetchSize": 1000,
    "maxIdleConns": 10,
    "idleTimeSec": 300,
    "queryTimeoutSec": 600,
    "propertyList": []
  }
}

Parameters

hostname Body String

Google BigQuery hostname. Expected value is https://www.googleapis.com/bigquery/v2.

port Body String

Google BigQuery port number. Expected value is 443.

projectId Body String

Google Cloud Project ID.

Example: Project_unique_id

authMode Body String

Authentication mode for the Google BigQuery connector. Expected value is SERVICE_ACCOUNT_KEYS.

Example: SERVICE_ACCOUNT_KEYS

clientEmail Body String

Google service account email address.

Example: service-account@your_project.iam.gserviceaccount.com

privateKey Body String Optional

Service account key associated with the supplied service account. The REST API expects the JSON object within the service account key JSON file.

Example:

{
 "type": "service_account",
 "project_id": "{project_id}",
 "private_key_id": "key-id",
 "private_key": "-----BEGIN PRIVATE KEY-----\nprivate-key\n-----END PRIVATE KEY-----\n",
 "client_email": "service-account-email",
 "client_id": "client-id",
 "auth_uri": "https://accounts.google.com/o/oauth2/auth",
 "token_uri": "https://accounts.google.com/o/oauth2/token",
 "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
 "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/service-account-email"
}

fetchSize Body Integer Optional

Number of records to fetch at one time. Default is 200. If set to 0, Dremio automatically decides how many records to fetch.

Example: 200

maxIdleConns Body Integer Optional

Maximum number of connections that can be idle at any given time. Default is 8.

Example: 8

idleTimeSec Body Integer Optional

Maximum time that a connection can remain idle before Dremio terminates it, in seconds. Default is 60.

Example: 60

queryTimeoutSec Body Integer Optional

Maximum time to allow for query execution, in seconds. When the query timeout expires, the connection returns to an idle state. Default is 0.

Example: 0

propertyList Body Array of Object Optional

Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.

Example:

[
  {
    "name": "deferPrepares",
    "value": "true"
  }
]

IBM Db2

Dremio source type: DB2
See Connect to IBM Db2 for additional information.

Config attributes for IBM Db2

{
  "config": {
    "database": "tpch",
    "hostname": "172.25.1.245",
    "username": "exampleuser",
    "password": "$DREMIO_EXISTING_VALUE$",
    "port": "50000",
    "fetchSize": 200,
    "maxIdleConns": 8,
    "idleTimeSec": 60,
    "queryTimeoutSec": 0,
    "propertyList": [
      {
        "name": "deferPrepares",
        "value": "true"
      }
    ]
  }
}

Parameters

database Body String

IBM Db2 database for Dremio to use.

Example: tpch

hostname Body String

IBM Db2 hostname.

Example: 172.25.1.245

username Body String

Username for authentication.

Example: exampleuser

password Body String

The password in plaintext. To keep the password secure, Dremio returns the password value as $DREMIO_EXISTING_VALUE$ in API responses.

Plaintext password values must be prefixed with data:, — for example, data:,<plaintext_password>. Plaintext password values that contain a colon (:) must be encoded in base64 format and prefixed with data:;base64, — for example, data:;base64,<base64_encoded_password>.

port Body String

IBM Db2 port number. Default is 50000.

Example: 50000

fetchSize Body Integer Optional

Number of records to fetch at one time. Default is 200. If set to 0, Dremio automatically decides how many records to fetch.

Example: 200

maxIdleConns Body Integer Optional

Maximum number of connections that can be idle at any given time. Default is 8.

Example: 8

idleTimeSec Body Integer Optional

Maximum time that a connection can remain idle before Dremio terminates it, in seconds. Default is 60.

Example: 60

queryTimeoutSec Body Integer Optional

Maximum time to allow for query execution, in seconds. When the query timeout expires, the connection returns to an idle state. Default is 0.

Example: 0

propertyList Body Array of Object Optional

Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.

Example:

[
  {
    "name": "deferPrepares",
    "value": "true"
  }
]

Iceberg REST Catalog

Dremio source type: RESTCATALOG
See Connect to an Iceberg REST Catalog for additional information.

Config attributes for an Iceberg REST Catalog

{
  "config": {
    "propertyList": [
      {
        "name": "warehouse",
        "value": "polaris_oss_catalog"
      },
      {
        "name": "scope",
        "value": "PRINCIPAL_ROLE:ALL"
      },
      {
        "name": "fs.s3a.aws.credentials.provider",
        "value": "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"
      }
    ],
    "secretPropertyList": [
      {
        "name": "fs.s3a.access.key",
        "value": "$DREMIO_EXISTING_VALUE$"
      },
      {
        "name": "fs.s3a.secret.key",
        "value": "$DREMIO_EXISTING_VALUE$"
      },
      {
        "name": "credential",
        "value": "client_id:client_secret"
      }
    ],
    "enableAsync": true,
    "isCachingEnabled": true,
    "maxCacheSpacePct": 100,
    "restEndpointUri": "http://localhost:8181/api/catalog",
    "allowedNamespaces": [
      "sales",
      "finance"
    ],
    "isUsingVendedCredentials": false,
    "isRecursiveAllowedNamespaces": true
  }
}

Parameters

propertyList Body Array of Object Optional

Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property. For a complete list of the catalog properties required for storage authentication, see Advanced Options on the Iceberg REST Catalog page.

The parameters needed will vary by the catalog to which you are connecting. The example above captures how to connect to a Polaris catalog using S3 storage and credentials. Examples of supported configurations can be found here: Supported Configurations.

Example:

[
  {
    "name": "fs.s3a.aws.credentials.provider",
    "value": "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"
  }
]

secretPropertyList Body Array of Object Optional

Example:

[
  {
    "name": "fs.s3a.access.key",
    "value": "$DREMIO_EXISTING_VALUE$"
  }
]

enableAsync Body Boolean Optional

To enable asynchronous access for the source and allow cloud caching to support simultaneous actions such as adding and editing, set to true (default). Otherwise, set to false.

isCachingEnabled Body Boolean Optional

To enable local caching, set to true (default). Otherwise, set to false.

maxCacheSpacePct Body Integer Optional

Maximum percentage of the total available cache space to use on any single executor node. Used only when isCachingEnabled is set to true. Minimum is 1. Maximum is 100 (default).

Example: 100

restEndpointUri Body String

URI for the catalog service that Dremio should connect to.

Example: http://localhost:8181/api/catalog

isUsingVendedCredentials Body Boolean

allowedNamespaces Body Array of String Optional

Example:

[
  "sales",
  "finance"
]

isRecursiveAllowedNamespaces Body Boolean Optional

If allowedNamespaces include their whole subtrees, set to true (default). Otherwise, set to false.

Parameters of Objects in the `propertyList` Array

name Body String Optional

Name of the connection property. For a complete list of the catalog properties required for storage authentication, see Advanced Options on the Iceberg REST Catalog page.

Example: fs.s3a.aws.credentials.provider

value Body String Optional

Value for the connection property. For a complete list of the catalog properties required for storage authentication, see Advanced Options on the Iceberg REST Catalog page.

Example: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider

Parameters of Objects in the `secretPropertyList` Array

name Body String Optional

Name of the connection property. For a complete list of the catalog credentials required for storage authentication, see Advanced Options on the Iceberg REST Catalog page.

Example: fs.s3a.secret.key

value Body String Optional

Value for the connection property. The value is masked in the Dremio console. For a complete list of the catalog credentials required for storage authentication, see Advanced Options on the Iceberg REST Catalog page.

Example: l1hDG6w6E1w1LgTrRwDwxNaiWV5wn18210536VhD

Microsoft Azure Synapse Analytics

Dremio source type: AZURESYNAPSE
See Connect to Microsoft Azure Synapse Analytics for additional information.

Config attributes for Microsoft Azure Synapse Analytics

{
  "config": {
    "hostname": "synapsedata.sql.azuresynapse.net",
    "port": "1433",
    "username": "exampleuser",
    "password": "$DREMIO_EXISTING_VALUE$",
    "authenticationType": "MASTER",
    "fetchSize": 200,
    "useSsl": false,
    "enableServerVerification": true,
    "maxIdleConns": 8,
    "idleTimeSec": 60,
    "queryTimeoutSec": 0,
    "database": "dedicatedpool",
    "propertyList": [
      {
        "name": "applicationIntent",
        "value": "ReadWrite"
      }
    ]
  }
}

Parameters

hostname Body String

Microsoft Azure Synapse Analytics hostname.

Example: synapsedata.sql.azuresynapse.net

port Body String Optional

Microsoft Azure Synapse Analytics port number. Default is 1433.

Example: 1433

username Body String

Username for authentication. Required for authenticationType MASTER.

Example: exampleuser

password Body String

The password for authentication. Required for authenticationType MASTER. To keep the password secure, Dremio returns the password value as $DREMIO_EXISTING_VALUE$ in API responses.

authenticationType Body String

Type of authentication to use to allow Dremio to connect to the Microsoft Azure Synapse Analytics source.

ANONYMOUS: No authentication is needed.
MASTER: Use credentials from a master database user.

Example: MASTER

fetchSize Body Integer Optional

Number of records to fetch at one time. Default is 200. If set to 0, Dremio automatically decides how many records to fetch.

Example: 200

useSsl Body Boolean Optional

If the Microsoft Azure Synapse Analytics source is configured to use TLS for connections, set to true. Otherwise, set to false (default).

enableServerVerification Body Boolean Optional

To verify the certificate for the Microsoft Azure Synapse Analytics source when connecting, set to true. Otherwise, set to false (default).

maxIdleConns Body Integer Optional

Maximum number of connections that can be idle at any given time. Default is 8.

Example: 8

idleTimeSec Body Integer Optional

Maximum time that a connection can remain idle before Dremio terminates it, in seconds. Default is 60.

Example: 60

queryTimeoutSec Body Integer Optional

Maximum time to allow for query execution, in seconds. When the query timeout expires, the connection returns to an idle state. Default is 0.

Example: 0

database Body String Optional

Default Microsoft Azure Synapse Analytics database for Dremio to use.

Example: dedicatedpool

propertyList Body Array of Object Optional

Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.

Example:

[
  {
    "name": "applicationIntent",
    "value": "ReadWrite"
  }
]

Parameters of Objects in the `propertyList` Array

name Body String Optional

Name of the connection property.

Example: applicationIntent

value Body String Optional

Value for the connection property.

Example: ReadWrite

Microsoft SQL Server

Dremio source type: MSSQL
See Connect to Microsoft SQL Server for additional information.

Config attributes for Microsoft SQL Server

"config": {
  "authenticationType": "MASTER",
  "username": "myUsername",
  "password": "myPassword",
  "hostname": "localhost",
  "port": "1433",
  "database": "my_database",
  "useSsl": true,
  "showOnlyConnectionDatabase": false,
  "fetchSize": 200,
  "maxIdleConns": 8,
  "idleTimeSec": 60,
  "enableServerVerification": true,
  "hostnameOverride": "localhost",
  "propertyList": [
    {
      "name": "applicationIntent",
      "value": "ReadWrite"
    }
  ]
}

Parameters

authenticationType Body String

Specifies the type of authentication to enable Dremio to connect to the source.

ANONYMOUS: No authentication is needed.
MASTER: Credentials from a master database user are required.

Example: MASTER

username Body String

The SQL Server username.

Example: myUsername

password Body String

The SQL Server password.

Example: myPassword

hostname Body String

The SQL Server hostname.

Example: localhost

port Body String

The SQL Server port number. The commonly used default port numbers for SQL Server are 1433 and 1434.

Example: 1433

database Body String Optional

The database name.

Example: my_database

useSsl Body Boolean

Option to enable SSL encryption. Set to true to enable this option.

showOnlyConnectionDatabase Body Boolean

Option to show only the initial database used for connecting. Set to true to enable this option.

fetchSize Body Integer

The number of records to fetch per request. Set to 0 to enable Dremio to automatically set the size.

Example: 200

maxIdleConns Body Integer

Sets the maximum number of idle connections to keep.

Example: 8

idleTimeSec Body Integer

Sets the idle time, in seconds, before a connection is evaluated for closure.

Example: 60

enableServerVerification Body Boolean

Option to verify the server certificate. Set to true to enable this option.

hostnameOverride Body String

This parameter can be used when the SSL/TLS server certificate contains a distinguished name. You can leave this blank if the hostname in the certificate matches the server name.

Example: localhost

propertyList Body Array of Object Optional

An array of name/value pairs defining optional connection properties. Use a comma-separated list to specify multiple name/value pairs.

Example:

[
  {
    "name": "applicationIntent",
    "value": "ReadWrite"
  }
]

Authentication Examples

No authentication required

{
  "authenticationType": "ANONYMOUS"
}

Master credentials

{
  "authenticationType": "MASTER",
  "username": "myUsername",
  "password": "myPassword"
}

MySQL

Dremio source type: MYSQL
See Connect to MySQL for additional information.

Config attributes for MySQL

"config": {
  "authenticationType": "MASTER",
  "username": "myUsername",
  "password": "myPassword",
  "hostname": "localhost",
  "port": "3306",
  "netWriteTimeout": 60,
  "fetchSize": 200,
  "maxIdleConns": 8,
  "idleTimeSec": 60,
  "propertyList": [
    {
      "name": "DataSource",
      "value": "sales2023"
    }
  ]
}

Parameters

authenticationType Body String

Specifies the type of authentication to enable Dremio to connect to the source. Use ANONYMOUS when no authentication is needed and MASTER when credentials from a master database user are required.

Valid Values: ANONYMOUS, MASTER

username Body String

The MySQL username.

Example: myUsername

password Body String

The MySQL password.

Example: myPassword

hostname Body String

The MySQL hostname.

Example: localhost

port Body String

The MySQL port number. The commonly used default port number for MySQL is 3306.

Example: 3306

netWriteTimeout Body Integer

The amount of time, in seconds, to wait for data from the server before aborting the connection.

Example: 60

fetchSize Body Integer

The number of records to fetch per request. Set to 0 to enable Dremio to automatically set the size.

Example: 200

maxIdleConns Body Integer

Sets the maximum number of idle connections to keep.

Example: 8

idleTimeSec Body Integer

Sets the idle time, in seconds, before a connection is evaluated for closure.

Example: 60

propertyList Body Array of Object Optional

An array of name/value pairs defining optional connection properties. Use a comma-separated list to specify multiple name/value pairs.

Example:

[
  {
    "name": "DataSource",
    "value": "sales2023"
  }
]

Authentication Examples

No authentication required

{
  "authenticationType": "ANONYMOUS"
}

Master credentials

{
  "authenticationType": "MASTER",
  "username": "myUsername",
  "password": "myPassword"
}

Oracle

Dremio source type: ORACLE
See Connect to Oracle for additional information.

Config attributes for Oracle

"config": {
  "authenticationType": "MASTER",
  "username": "myUsername",
  "password": "myPassword",
  "hostname": "localhost",
  "port": "1521",
  "instance": "sales",
  "useSsl": true,
  "nativeEncryption": "ACCEPTED",
  "useTimezoneAsRegion": true,
  "includeSynonyms": false,
  "mapDateToTimestamp": true,
  "fetchSize": 200,
  "maxIdleConns": 8,
  "idleTimeSec": 60,
  "useLdap": false,
  "bindDN": "",
  "sslServerCertDN": "",
  "propertyList": [
    {
      "name": "connect_timeout",
      "value": "0"
    }
  ]
}

Parameters

authenticationType Body String

Specifies the type of authentication to enable Dremio to connect to the source.

ANONYMOUS: No authentication is needed.
MASTER: Credentials from a master database user or a secret resource URL are required.
KERBEROS: Kerberos is configured.

username Body String

The Oracle username.

Example: myUsername

password Body String

The Oracle password.

Example: myPassword

secretResourceUrl Body String

For secret-based authentication, the Amazon Resource Name (ARN) for the secret. To keep the secret secure, Dremio returns the secretResourceUrl value as $DREMIO_EXISTING_VALUE$ in API responses.

Example: arn:aws:secretsmanager:us-west-2:123456789012:secret:my-rds-secret-VNenFy

useKerberos Body Boolean

Set to true if you are using Kerberos.

hostname Body String

The Oracle hostname.

Example: localhost

port Body String

The Oracle port number. The commonly used default port numbers for Oracle are 1521 and 1830.

Example: 1521

instance Body String Optional

The Oracle service name.

Example: sales

useSsl Body Boolean

Option to enable SSL encryption. Set to true to enable this option.

nativeEncryption Body String

Sets the Oracle native encryption. This parameter can only be used if useSsl is set to false.

Valid Values: REJECTED, ACCEPTED, REQUESTED, REQUIRED

useTimezoneAsRegion Body Boolean

Option to use the timezone to set the connection region. Set to true to enable this option.

includeSynonyms Body Boolean

Option to include synonyms as datasets. Set to true to enable this option.

mapDateToTimestamp Body Boolean

Option to display Oracle DATE columns as TIMESTAMP. Set to true to enable this option.

fetchSize Body Integer

The number of records to fetch per request. Set to 0 to enable Dremio to automatically set the size.

Example: 200

maxIdleConns Body Integer

Sets the maximum number of idle connections to keep.

Example: 8

idleTimeSec Body Integer

Sets the idle time, in seconds, before a connection is evaluated for closure.

Example: 60

useLdap Body Boolean

Option to use LDAP naming services. Set to true to enable this option.

bindDN Body String

Sets the LDAP hostname resolution when the useLdap parameter is set to true. You can omit this parameter if useLdap is set to false.

sslServerCertDN Body String

Use when you have an SSL/TLS server certificate distinguished name. Otherwise, you can leave this parameter blank to disable DN match.

propertyList Body Array of Object Optional

An array of name/value pairs defining optional connection properties. Use a comma-separated list to specify multiple name/value pairs.

Example:

[
  {
    "name": "connect_timeout",
    "value": "0"
  }
]

Authentication Examples

No authentication required

{
  "authenticationType": "ANONYMOUS"
}

Master credentials

{
  "authenticationType": "MASTER",
  "username": "myUsername",
  "password": "myPassword"
}

Secret resource URL

{
  "authenticationType": "MASTER",
  "username": "myUsername",
  "secretResourceUrl": "arn:aws:secretsmanager:us-west-2:123456789012:secret:my-rds-secret-VNenFy"
}

Using Kerberos

{
  "authenticationType": "KERBEROS",
  "username": null,
  "password": null,
  "useKerberos": true
}

PostgreSQL

Dremio source type: POSTGRES
See Connect to PostgreSQL for additional information.

Config attributes for PostgreSQL

"config": {
  "authenticationType": "MASTER",
  "username": "myUsername",
  "password": "myPassword",
  "hostname": "localhost",
  "port": "5432",
  "databaseName": "sales_2023",
  "useSsl": false,
  "fetchSize": 200,
  "maxIdleConns": 8,
  "idleTimeSec": 60,
  "encryptionValidationMode": "CERTIFICATE_AND_HOSTNAME_VALIDATION",
  "propertyList": [
    {
      "name": "connect_timeout",
      "value": "10&application_name=myapp"
    }
  ]
}

Parameters

authenticationType Body String

Specifies the type of authentication to enable Dremio to connect to the source.

ANONYMOUS: No authentication is needed.
MASTER: Credentials from a master database user or a secret resource URL are required.

username Body String

The PostgreSQL username.

Example: myUsername

password Body String

The PostgreSQL password.

Example: myPassword

secretResourceUrl Body String

For secret-based authentication, the Amazon Resource Name (ARN) for the secret. To keep the secret secure, Dremio returns the secretResourceUrl value as $DREMIO_EXISTING_VALUE$ in API responses.

Example: arn:aws:secretsmanager:us-west-2:123456789012:secret:my-rds-secret-VNenFy

hostname Body String

The PostgreSQL hostname.

Example: localhost

port Body String

The PostgreSQL port number. The commonly used default port number for PostgreSQL is 5432.

Example: 5432

databaseName Body String Optional

The database name.

Example: sales_2023

useSsl Body Boolean

Option to enable SSL encryption. Set to true to enable this option.

fetchSize Body Integer

The number of records to fetch per request. Set to 0 to enable Dremio to automatically set the size.

Example: 200

maxIdleConns Body Integer

Sets the maximum number of idle connections to keep.

Example: 8

idleTimeSec Body Integer

Sets the idle time, in seconds, before a connection is evaluated for closure.

Example: 60

encryptionValidationMode Body String

When encryption is enabled, you can set the validation mode to one of the available presets.

Valid Values: CERTIFICATE_AND_HOSTNAME_VALIDATION, CERTIFICATE_ONLY_VALIDATION, NO_VALIDATION

propertyList Body Array of Object Optional

An array of name/value pairs defining optional connection properties. Use a comma-separated list to specify multiple name/value pairs.

Example:

[
  {
    "name": "connect_timeout",
    "value": "10&application_name=myapp"
  }
]

Authentication Examples

No authentication required

{
  "authenticationType": "ANONYMOUS"
}

Master credentials

{
  "authenticationType": "MASTER",
  "username": "myUsername",
  "password": "myPassword"
}

SAP HANA

Dremio source type: SAPHANA
See Configure SAP HANA as a Source for additional information.

Config attributes for SAP HANA

"config": {
  "hostname": "sap://hanaserver1.company.com",
  "port": "39017",
  "schema": "marketing",
  "username": "myUsername",
  "password": "myPassword",
  "maxIdleConns": 8,
  "idleTimeSec": 60,
  "queryTimeoutSec": 0,
  "fetchSize": 2000,
  "propertyList": []
}

Parameters

hostname Body String

The SAP HANA hostname. This can be a fully qualified domain name or an IP address.

Example: sap://hanaserver1.company.com

port Body String

The SAP HANA port number. The commonly used default port number for SAP HANA is 39017.

Example: 39017

schema Body String Optional

Specifies the schema the connector will use.

Example: marketing

username Body String

Username for authentication.

Example: myUsername

password Body String

The SAP HANA password.

Example: myPassword

fetchSize Body Integer

The number of records to fetch per request. Set to 0 to enable Dremio to automatically set the size.

Example: 2000

maxIdleConns Body Integer

Sets the maximum number of idle connections to keep.

Example: 8

idleTimeSec Body Integer

Sets the idle time, in seconds, before a connection is evaluated for closure.

Example: 60

queryTimeoutSec Body Integer

Sets the timeout, in seconds, for query execution before it is canceled. If you do not want a timeout, enter 0.

Example: 0

propertyList Body Array of Object Optional

Connection properties for the source. Each object in propertyList includes the name of the property and the corresponding value to use for the property. For SAP HANA, see the list of supported JDBC Connection Properties.

Example:

[
  {
    "name": "connectTimeout",
    "value": "0"
  }
]

Snowflake

Dremio source type: SNOWFLAKE
See Configure Snowflake as a Source for additional information.

Config attributes for Snowflake

"config": {
  "hostname": "https://locator_id.snowflakecomputing.com",
  "port": "443",
  "database": "REGRESSION",
  "warehouse": "CONNECTOR_25GB",
  "username": "dremio",
  "maxIdleConns": 8,
  "idleTimeSec": 60,
  "queryTimeoutSec": 0,
  "fetchSize": 2000,
  "authMode": "KEY_PAIR",
  "privateKey": "AsB86vDRA&Ew5+jSCwn3F2LqMcmu82DcN9ByfcDzcP2GE2oJivFAS53M9s4B=hnu5Eg"
}

Parameters

hostname Body String

The Snowflake hostname.

Example: https://locator_id.snowflakecomputing.com

port Body String

The Snowflake port number. The commonly used default port number for Snowflake is 443.

Example: 443

database Body String Optional

Specifies the default database to use.

Example: REGRESSION

warehouse Body String Optional

Specifies the warehouse that will provide resources for executing DML statements and queries.

Example: CONNECTOR_25GB

username Body String

The Snowflake username.

Example: dremio

password Body String

The Snowflake password. Required if authMode is LOGIN_PASSWORD.

Example: myPassword

fetchSize Body Integer

The number of records to fetch per request. Set to 0 to enable Dremio to automatically set the size.

Example: 2000

maxIdleConns Body Integer

Sets the maximum number of idle connections to keep.

Example: 8

idleTimeSec Body Integer

Sets the idle time, in seconds, before a connection is evaluated for closure.

Example: 60

queryTimeoutSec Body Integer

Sets the timeout, in seconds, for query execution before it is canceled. If you do not want a timeout, enter 0.

Example: 0

authMode Body String

Type of authentication for Dremio to use to connect to the source. Must be LOGIN_PASSWORD (default) or KEY_PAIR.

privateKey Body String

The generated Snowflake private key in Privacy Enhanced Mail (PEM) format. Required if authMode is KEY_PAIR. To keep the service account key secure, Dremio returns the privateKey value as $DREMIO_EXISTING_VALUE$ in API responses.

Example: AsB86vDRA&Ew5+jSCwn3F2LqMcmu82DcN9ByfcDzcP2GE2oJivFAS53M9s4B=hnu5Eg

privateKeyPassphrase Body String

The passphrase for an encrypted private key. Required if authMode is KEY_PAIR and the privateKey value is an encrypted private key.

Example: !jV#X=%k7cG=e%G64p#ozm

Authentication Examples

{
  "authenticationType": "LOGIN_PASSWORD",
  "username": "dremio",
  "password": "myPassword"
}

Key-pair authentication

{
  "authenticationType": "KEY_PAIR",
  "username": "dremio",
  "privateKey": "AsB86vDRA&Ew5+jSCwn3F2LqMcmu82DcN9ByfcDzcP2GE2oJivFAS53M9s4B=hnu5Eg",
  "privateKeyPassphrase": "!jV#X=%k7cG=e%G64p#ozm"
}

Snowflake Open Catalog

Dremio source type: SNOWFLAKEOPENCATALOG
See Configure Snowflake Open Catalog for additional information.

Config attributes for Snowflake Open Catalog

{
  "config": {
    "propertyList": [
      {
        "name": "fs.s3a.aws.credentials.provider",
        "value": "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"
      }
    ],
    "secretPropertyList": [
      {
        "name": "fs.s3a.access.key",
        "value": "$DREMIO_EXISTING_VALUE$"
      },
      {
        "name": "fs.s3a.secret.key",
        "value": "$DREMIO_EXISTING_VALUE$"
      }
    ],
    "enableAsync": true,
    "isCachingEnabled": true,
    "maxCacheSpacePct": 100,
    "restEndpointUri": "https://phy87032.snowflakecomputing.com/polaris/api/catalog",
    "allowedNamespaces": [
      "sales",
      "finance"
    ],
    "isUsingVendedCredentials": false,
    "isRecursiveAllowedNamespaces": true,
    "snowflakeOpenCatalogWarehouse": "snowflake-open-catalog",
    "snowflakeOpenCatalogClientId": "ImN1IMc4SvYjkB63X3aj0P8w7H86",
    "snowflakeOpenCatalogClientSecret": "$DREMIO_EXISTING_VALUE$",
    "snowflakeOpenCatalogScope": "PRINCIPAL_ROLE:ALL"
  }
}

Parameters

propertyList Body Array of Object Optional

Example:

[
  {
    "name": "fs.s3a.aws.credentials.provider",
    "value": "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"
  }
]

secretPropertyList Body Array of Object Optional

Additional connection properties for the source. The values are kept masked in the Credentials section of the Advanced Options page of the Source Settings dialog in the Dremio console. Each object includes the name of the property and the corresponding value to use and to keep secret. To keep the secret secure, Dremio returns the secretPropertyList value as $DREMIO_EXISTING_VALUE$ in API responses. For a complete list of the catalog credentials required for storage authentication, see Advanced Options on the Snowflake Open Catalog page.

Example:

[
  {
    "name": "fs.s3a.access.key",
    "value": "$DREMIO_EXISTING_VALUE$"
  }
]

enableAsync Body Boolean Optional

To enable asynchronous access for the source and allow cloud caching to support simultaneous actions such as adding and editing, set to true (default). Otherwise, set to false.

isCachingEnabled Body Boolean Optional

To enable local caching, set to true (default). Otherwise, set to false.

maxCacheSpacePct Body Integer Optional

Maximum percentage of the total available cache space to use on any single executor node. Used only when isCachingEnabled is set to true. Minimum is 1. Maximum is 100 (default).

Example: 100

restEndpointUri Body String

URI for the catalog service that Dremio should connect to.

Example: https://phy87032.snowflakecomputing.com/polaris/api/catalog

isUsingVendedCredentials Body Boolean

allowedNamespaces Body Array of String Optional

Example:

[
  "sales",
  "finance"
]

isRecursiveAllowedNamespaces Body Boolean Optional

If allowedNamespaces include their whole subtrees, set to true (default). Otherwise, set to false.

snowflakeOpenCatalogWarehouse Body String

The Snowflake Open Catalog name.

Example: snowflake-open-catalog

snowflakeOpenCatalogClientId Body String

The client ID created during the configuration of a service connection for the Snowflake Open Catalog.

Example: ImN1IMc4SvYjkB63X3aj0P8w7H86

snowflakeOpenCatalogClientSecret Body String

The client secret in plaintext. To keep the client secret secure, Dremio returns the snowflakeOpenCatalogClientSecret value as $DREMIO_EXISTING_VALUE$ in API responses.

Plaintext snowflakeOpenCatalogClientSecret values must be prefixed with data:, — for example, data:,<plaintext_client_secret>. Plaintext snowflakeOpenCatalogClientSecret values that contain a colon (:) must be encoded in base64 format and prefixed with data:;base64, — for example, data:;base64,<base64_encoded_client_secret>.

snowflakeOpenCatalogScope Body String Optional

The OAuth 2.0 scope limit specifying the principal role that is allowed access to the Snowflake Open Catalog.

Example: PRINCIPAL_ROLE:ALL

Parameters of Objects in the `propertyList` Array

name Body String Optional

Name of the connection property. For a complete list of the catalog properties required for storage authentication, see Advanced Options on the Snowflake Open Catalog page.

Example: fs.s3a.aws.credentials.provider

value Body String Optional

Value for the connection property. For a complete list of the catalog properties required for storage authentication, see Advanced Options on the Snowflake Open Catalog page.

Example: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider

Parameters of Objects in the `secretPropertyList` Array

name Body String Optional

Name of the connection property. For a complete list of the catalog credentials required for storage authentication, see Advanced Options on the Snowflake Open Catalog page.

Example: fs.s3a.secret.key

value Body String Optional

Example: l1hDG6w6E1w1LgTrRwDwxNaiWV5wn18210536VhD

Unity Catalog

Dremio source type: UNITY
See Connect to a Databricks Unity Catalog for additional information.

Config attributes for Unity Catalog

{
  "config": {
    "propertyList": [
      {
        "name": "fs.s3a.aws.credentials.provider",
        "value": "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"
      }
    ],
    "secretPropertyList": [
      {
        "name": "fs.s3a.access.key",
        "value": "$DREMIO_EXISTING_VALUE$"
      },
      {
        "name": "fs.s3a.secret.key",
        "value": "$DREMIO_EXISTING_VALUE$"
      }
    ],
    "enableAsync": true,
    "isCachingEnabled": true,
    "maxCacheSpacePct": 100,
    "restEndpointUri": "https://dbc-3i6c5lto-k3v3.cloud.databricks.com/api/2.1/unity-catalog/iceberg",
    "allowedNamespaces": [
      "sales",
      "finance"
    ],
    "isUsingVendedCredentials": false,
    "isRecursiveAllowedNamespaces": true,
    "unityAuthToken": "$DREMIO_EXISTING_VALUE$",
    "unityCatalog": "unity_catalog"
  }
}

Parameters

propertyList Body Array of Object Optional

Example:

[
  {
    "name": "fs.s3a.aws.credentials.provider",
    "value": "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"
  }
]

secretPropertyList Body Array of Object Optional

Additional connection properties for the source. The values are kept masked in the Credentials section of the Advanced Options page of the Source Settings dialog in the Dremio console. Each object includes the name of the property and the corresponding value to use and to keep secret. To keep the secret secure, Dremio returns the secretPropertyList value as $DREMIO_EXISTING_VALUE$ in API responses. For a complete list of the catalog credentials required for storage authentication, see Advanced Options on the Unity Catalog page.

Example:

[
  {
    "name": "fs.azure.account.key",
    "value": "70SPup32UsIZaA6c2n6bf3rQONTD6Zn6OqvcUhBy"
  }
]

enableAsync Body Boolean Optional

To enable asynchronous access for the source and allow cloud caching to support simultaneous actions such as adding and editing, set to true (default). Otherwise, set to false.

isCachingEnabled Body Boolean Optional

To enable local caching, set to true (default). Otherwise, set to false.

maxCacheSpacePct Body Integer Optional

Maximum percentage of the total available cache space to use on any single executor node. Used only when isCachingEnabled is set to true. Minimum is 1. Maximum is 100 (default).

Example: 100

restEndpointUri Body String

URI for the catalog service that Dremio should connect to.

Example: https://dbc-3i6c5lto-k3v3.cloud.databricks.com/api/2.1/unity-catalog/iceberg

isUsingVendedCredentials Body Boolean

allowedNamespaces Body Array of String Optional

Example:

[
  "sales",
  "finance"
]

isRecursiveAllowedNamespaces Body Boolean Optional

If allowedNamespaces include their whole subtrees, set to true (default). Otherwise, set to false.

unityAuthToken Body String

The authentication token in plaintext. To keep the token secure, Dremio returns the unityAuthToken value as $DREMIO_EXISTING_VALUE$ in API responses.

Plaintext unityAuthToken values must be prefixed with data:, — for example, data:,<plaintext_token>. Plaintext unityAuthToken values that contain a colon (:) must be encoded in base64 format and prefixed with data:;base64, — for example, data:;base64,<base64_encoded_token>.

unityCatalog Body String

Name of the Unity Catalog.

Example: unity_catalog

Parameters of Objects in the `propertyList` Array

name Body String Optional

Name of the connection property. For a complete list of the catalog properties required for storage authentication, see Advanced Options on the Unity Catalog page.

Example: fs.s3a.aws.credentials.provider

value Body String Optional

Value for the connection property. For a complete list of the catalog properties required for storage authentication, see Advanced Options on the Unity Catalog page.

Example: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider

Parameters of Objects in the `secretPropertyList` Array

name Body String Optional

Name of the connection property. For a complete list of the catalog credentials required for storage authentication, see Advanced Options on the Unity Catalog page.

Example: fs.s3a.access.key

value Body String Optional

Example: e0820ta9vgZlhOB9P9VABCbp7bsd8vP4Miqxisk5BUq9ZrqbufF1QvA5b99m0848ulmTbAUwG2wNZUY1vr8yoiKe

Vertica

Dremio source type: VERTICA
See Connect to Vertica for additional information.

Config attributes for Vertica

{
  "config": {
    "database": "vertica",
    "hostname": "172.25.0.10",
    "username": "exampleuser",
    "password": "$DREMIO_EXISTING_VALUE$",
    "port": "5433",
    "fetchSize": 200,
    "maxIdleConns": 8,
    "idleTimeSec": 60,
    "queryTimeoutSec": 0,
    "propertyList": [
      {
        "name": "LoginTimeout",
        "value": "30"
      }
    ]
  }
}

Parameters

database Body String

Default Vertica database for Dremio to use.

Example: vertica

hostname Body String

Vertica hostname.

Example: 172.25.0.10

username Body String

Username for authentication.

Example: exampleuser

password Body String

The password in plaintext. To keep the password secure, Dremio returns the password value as $DREMIO_EXISTING_VALUE$ in API responses.

port Body String

Vertica port number. Default is 5433.

Example: 5433

fetchSize Body Integer Optional

Number of records to fetch at one time. Default is 200. If set to 0, Dremio automatically decides how many records to fetch.

Example: 200

maxIdleConns Body Integer Optional

Maximum number of connections that can be idle at any given time. Default is 8.

Example: 8

idleTimeSec Body Integer Optional

Maximum time that a connection can remain idle before Dremio terminates it, in seconds. Default is 60.

Example: 60

queryTimeoutSec Body Integer Optional

Maximum time to allow for query execution, in seconds. When the query timeout expires, the connection returns to an idle state. Default is 0.

Example: 0

propertyList Body Array of Object Optional

Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.

Example:

[
  {
    "name": "LoginTimeout",
    "value": "30&application_name=myapp"
  }
]

Parameters of Objects in the `propertyList` Array

name Body String Optional

Name of the connection property.

Example: LoginTimeout

value Body String Optional

Value for the connection property.

Example: 30

Amazon Redshift​

Parameters​

Authentication Examples​

Amazon S3​

Parameters​

Authentication Examples​

AWS Glue Data Catalog​

Parameters​

Authentication Examples​

Open Catalog​

Parameters​

Google Cloud Lakehouse Catalog​

Parameters​

Parameters of Objects in the propertyList Array​

Parameters of Objects in the secretPropertyList Array​

Google BigQuery​

Parameters​

IBM Db2​

Parameters​

Iceberg REST Catalog​

Parameters​

Parameters of Objects in the propertyList Array​

Parameters of Objects in the secretPropertyList Array​

Microsoft Azure Synapse Analytics​

Parameters​

Parameters of Objects in the propertyList Array​

Microsoft SQL Server​

Parameters​

Authentication Examples​

MySQL​

Parameters​

Authentication Examples​

Oracle​

Parameters​

Authentication Examples​

PostgreSQL​

Parameters​

Authentication Examples​

SAP HANA​

Parameters​

Snowflake​

Parameters​

Authentication Examples​

Snowflake Open Catalog​

Parameters​

Parameters of Objects in the propertyList Array​

Parameters of Objects in the secretPropertyList Array​

Unity Catalog​

Parameters​

Parameters of Objects in the propertyList Array​

Parameters of Objects in the secretPropertyList Array​

Vertica​

Parameters​

Parameters of Objects in the propertyList Array​

Amazon Redshift

Parameters

Authentication Examples

Amazon S3

Parameters

Authentication Examples

AWS Glue Data Catalog

Parameters

Authentication Examples

Open Catalog

Parameters

Google Cloud Lakehouse Catalog

Parameters

Parameters of Objects in the `propertyList` Array

Parameters of Objects in the `secretPropertyList` Array

Google BigQuery

Parameters

IBM Db2

Parameters

Iceberg REST Catalog

Parameters

Parameters of Objects in the `propertyList` Array

Parameters of Objects in the `secretPropertyList` Array

Microsoft Azure Synapse Analytics

Parameters

Parameters of Objects in the `propertyList` Array

Microsoft SQL Server

Parameters

Authentication Examples

MySQL

Parameters

Authentication Examples

Oracle

Parameters

Authentication Examples

PostgreSQL

Parameters

Authentication Examples

SAP HANA

Parameters

Snowflake

Parameters

Authentication Examples

Snowflake Open Catalog

Parameters

Parameters of Objects in the `propertyList` Array

Parameters of Objects in the `secretPropertyList` Array

Unity Catalog

Parameters

Parameters of Objects in the `propertyList` Array

Parameters of Objects in the `secretPropertyList` Array

Vertica

Parameters

Parameters of Objects in the `propertyList` Array