Welcome to Dremio Cloud Documentation - Dremio > > > >

Configuration Attributes

This page details the config payload to create each source type.

Amazon Redshift

Dremio source type: REDSHIFT. This type determines the config parameters that display as specified in the Source Attributes. If you are not familiar with the following parameters that are available for Redshift, see Amazon Redshift Configuration for additional information.

Config attributes for connecting to an Amazon Redshift source

"config": {
  "connectionString": "jdbc:redshift://redshift.example.com:5439/database",
  "authenticationType": [
    "ANONYMOUS",
    "MASTER",
    "AWS_PROFILE"
  ],
  "username": "myUsername",
  "password": "myPassword",
  "secretResourceUrl": arn:aws:secretsmanager:us-west-2:123456789012:secret:my-rds-secret-VNenFy,
  "awsProfile": "profileName",
  "dbUser": "userName",
  "fetchSize": 200,
  "maxIdleConns": 8,
  "idleTimeSec": 60,
  "queryTimeoutSec": 0,
  "propertyList": [
    {
      "name": "ApplicationName", 
      "value": "MarketingDashboard"
    }
  ]
}

Parameters

The JDBC connection string. For more information, read JDBC.

Specifies the type of authentication to enable Dremio to connect to the source. Use ANONYMOUS when no authentication is needed, MASTER when either credentials from a master database user or a secret resource URL is required, or AWS_PROFILE for authentication to an AWS source.

Enum ANONYMOUS , AWS_PROFILE , MASTER

The Amazon Redshift user name.

Example myUsername

The Amazon Redshift password.

Example myPassword

The secret resource URL.

The AWS profile name. If this is left blank, the default profile will be used.

Example profileName

The name of the Amazon Redshift DbUser that will be used for authentication. If this is left blank, the default user name for your AWS IAM role will be used.

Example userName

The number of records to fetch per request. Set to 0 to enable Dremio to automatically set the size.

Example 200

Sets the maximum number of idle connections that you want to keep.

Example 8

Sets the idle time, in seconds, before a connection is evaluated for closure.

Example 60

Sets the timeout, in seconds, for a query execution before it is canceled. if you do not want a timeout, enter 0.

An array of name/value pairs defining optional connection properties. Use a comma separated list to specify multiple name value pairs.

Example "name": "ApplicationName", "value": "MarketingDashboard"

Examples of Different Configurations for authenticationType

No authentication required

{
  "authenticationType": "ANONYMOUS"
}

Master credentials

{
  "authenticationType": "MASTER",
  "username": "myUsername",
  "password": "myPassword"
}

Secret resource URL

{
  "authenticationType": "MASTER",
  "username": "myUsername",
  "secretResourceUrl": "myPassword"
}

AWS Profile

{
  "authenticationType": "AWS_PROFILE",
  "awsProfile": "profileName",
  "dbUser": "userName"
}

Amazon S3

Dremio source type: S3. This type determines the config parameters that display as specified in the Source Attributes. If you are not familiar with the following parameters that are available for S3, see Adding an S3 Source for additional information.

Config attributes for connecting to an Amazon S3 source

"config": {
  "credentialType": [
    "PROJECT_DATA_CRED",
    "ACCESS_KEY"
  ]
  "assumedRoleARN": "arn:aws:iam::123456789012:root",
  "awsAccessKey": "AKIAIOSFODNN7EXAMPLE",
  "awsAccessSecret": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
  "externalBucketList": /bucket-name/,
  "secure": false,
  "enableAsync": true,
  "compatibilityMode": false,
  "requesterPays": false,
  "enableFileStatusCheck": true,
  "isPartitionInferenceEnabled": false,
  "rootPath": "/",
  "kmsKeyARN": "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab",
  "defaultCtasFormat": "ICEBERG",
  "propertyList": [
    {
      "name": "fs.s3a.proxy.host", 
      "value": "proxyHost.example.com"
    },
  ],
  "whitelistedBuckets": "sales",
  "isCachingEnabled": true,
  "maxCacheSpacePct": 100
}

Parameters

Specifies the type of credential to enable Dremio to connect to the source. Must be either ACCESS_KEY or PROJECT_DATA_CRED.

Enum ACCESS_KEY , PROJECT_DATA_CRED

The ARN of the IAM role to assume.

Example arn:aws:iam::123456789012:root

The AWS access key.

Example AKIAIOSFODNN7EXAMPLE

The AWS access secret.

Example wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Connects Dremio to one or more Amazon S3 buckets. For more information, read Accessing a bucket using S3.

Example /bucket-name/

Option to enable a secure connection via SSL encryption between the S3 buckets and Dremio. Set to true to enable SSL encryption.

Option to enable asynchronous access, which allows cloud caching for the source to support simultaneous actions such as adding and editing. Set to true to enable asynchronous access. For more information, read Amazon S3 > Advanced Options.

Option to enable compatibility mode for an Amazon S3 source, which allows you to use S3-compatible storage. Set to true to enable compatibility mode.

Option to apply S3 requests to the requestor. Set to true enable this option.

Option to enable file status check. Set to true to enable file status check.

Option to enable partition column inference. Set to true to enable partition column inference.

The root path of the S3 bucket.

Example /

Enables you to add a server side encryption key ARN that was created in AWS Key Management Service (KMS).

Example arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab

Enables you to choose the default format for tables you create in Dremio, either ICEBERG or PARQUET.

Enum ICEBERG , PARQUET

An array of name/value pairs defining optional connection properties. Use a comma separated list to specify multiple name value pairs.

Example "name": "fs.s3a.proxy.host", "value": "proxyHost.example.com"

Add one or more approved S3 buckets.

Example sales

Option to enable local caching. When local caching is possible, set this parameter to true to enable it. For more information, read Amazon S3 > Advanced Options.

Sets the maximum percent of total available cache space to use when possible. The value, in percent, must be between 1 and 100.

Example 100

Examples of Different Configurations for credentialType

Project data credentials

{
  "credentialType": "PROJECT_DATA_CRED"
}

Data source credentials (IAM role)

{
  "credentialType": "PROJECT_DATA_CRED",
  "assumedRoleARN": "arn:aws:iam::123456789012:root"
}

Data source credentials (access key)

{
  "credentialType": "ACCESS_KEY",
  "assumedRoleARN": "arn:aws:iam::123456789012:root",
  "awsAccessKey": "AKIAIOSFODNN7EXAMPLE",
  "awsAccessSecret": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
}

Arctic (Preview)

Dremio source type: ARCTIC. This type determines the config parameters that display as specified in the Source Attributes. If you are not familiar with the following parameters that are available for Arctic, see Dremio Sonar for additional information.

Config attributes for configuring an Arctic source

"config": {
  "awsRootPath": "/your-s3-bucket-name",
  "asyncEnabled": true,
  "isCachingEnabled": true,
  "maxCacheSpacePct": 100,
  "defaultCtasFormat": "ICEBERG",
  "credentialType": [
    "PROJECT_DATA_CRED",
    "ACCESS_KEY"
  ]
  "assumedRoleARN": "arn:aws:iam::123456789012:root",
  "arcticCatalogId": "ab12cdef-a123-b456-ab1234-cdef5678ghij"
},

Parameters

The root path of the Amazon S3 bucket.

Example /your-s3-bucket-name

Option to enable asynchronous access, when this functionality is available in your Arctic source. This option enables cloud caching for the source to support simultaneous actions such as adding and editing. Set to true to enable asynchronous access.

Option to enable local caching, when this functionality is available in your Arctic source. Set to true to enable local caching.

Sets the maximum percent of total available cache space to use when possible. The value, in percent, must be between 1 and 100.

Example 100

Enables you to choose the default format for tables you create in Dremio, either ICEBERG or PARQUET.

Enum ICEBERG , PARQUET

Specifies the type of credential to enable Dremio to connect to the source.

Enum ACCESS_KEY , PROJECT_DATA_CRED

The ARN of the IAM role to assume.

Example arn:aws:iam::123456789012:root

The AWS access key.

Example AKIAIOSFODNN7EXAMPLE

The AWS access secret.

Example wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

The unique ID of the Arctic catalog you are aceessing.

Example ab12cdef-a123-b456-ab1234-cdef5678ghij

Examples of Different Configurations for credentialType

Project data credentials

{
  "credentialType": "PROJECT_DATA_CRED"
}

Data source credentials (IAM role)

{
  "credentialType": "PROJECT_DATA_CRED",
  "assumedRoleARN": "arn:aws:iam::123456789012:root"
}

Data source credentials (access key)

{
  "credentialType": "ACCESS_KEY",
  "assumedRoleARN": "arn:aws:iam::123456789012:root",
  "awsAccessKey": "AKIAIOSFODNN7EXAMPLE",
  "awsAccessSecret": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
}

AWS Glue Data Catalog

Dremio source type: AWSGLUE. This type determines the config parameters that display as specified in the Source Attributes. If you are not familiar with the following parameters that are available for AWS Glue, see Adding an AWS Glue Data Catalog for additional information.

Config attributes for connecting to an AWS Glue Data Catalog

"config": {
  "credentialType": [
    "PROJECT_DATA_CRED",
    "ACCESS_KEY"
  ]
  "assumedRoleARN": "arn:aws:iam::123456789012:root",
  "awsAccessKey": "AKIAIOSFODNN7EXAMPLE",
  "awsAccessSecret": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
  "regionNameSelection": "us-east-1",
  "secure": false,
  "enableAsync": true,
  "propertyList": [
   {
      "name": "hive.metastore.warehouse.dir", 
      "value": "s3://bucket-name/folder"
    },
  ],
  "isCachingEnabled": true,
  "maxCacheSpacePct": 100
}

Parameters

Specifies the type of credential to enable Dremio to connect to the source. Must be either ACCESS_KEY or PROJECT_DATA_CRED.

Enum ACCESS_KEY , PROJECT_DATA_CRED

The ARN of the IAM role to assume.

Example arn:aws:iam::123456789012:root

The AWS access key.

Example AKIAIOSFODNN7EXAMPLE

The AWS access secret.

Example wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Sets the AWS Region where the Glue Data Catalog is located.

Example us-east-1, eu-west-2

Option to enable a secure connection via SSL encryption between the Glue Data Catalog and Dremio. Set to true to enable SSL encryption.

An array of name/value pairs defining optional connection properties. Use a comma separated list to specify multiple name value pairs.

Example "name": "hive.metastore.warehouse.dir", "value": "s3://bucket-name/folder"

Option to enable local caching. When local caching is possible, set this parameter to true to enable it. For more information, read AWS Glue Data Catalog > Advanced Options.

Sets the maximum percent of total available cache space to use when possible. The value, in percent, must be between 1 and 100.

Example 100

Examples of Different Configurations for credentialType

Project data credentials

{
  "credentialType": "PROJECT_DATA_CRED"
}

Data source credentials (IAM role)

{
  "credentialType": "PROJECT_DATA_CRED",
  "assumedRoleARN": "arn:aws:iam::123456789012:root"
}

Data source credentials (access key)

{
  "credentialType": "ACCESS_KEY",
  "assumedRoleARN": "arn:aws:iam::123456789012:root",
  "awsAccessKey": "AKIAIOSFODNN7EXAMPLE",
  "awsAccessSecret": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
}

Microsoft SQL Server

Dremio source type: MSSQL. This type determines the config parameters that display as specified in the Source Attributes. If you are not familiar with the following parameters that are available for SQL Server, see SQL Server Configuration for additional information.

Config attributes for connecting to a Microsoft SQL Server source

"config": {
  "authenticationType": [
    "ANONYMOUS",
    "MASTER"
  ]
  "username": "myUsername",
  "password": "myPassword".
  "hostname": "localhost",
  "port": "1433",
  "database": "my_database",
  "useSsl": true,
  "showOnlyConnectionDatabase": false,
  "fetchSize": 200,
  "maxIdleConns": 8,
  "idleTimeSec": 60,
  "enableServerVerification": true,
  "hostnameOverride": "localhost",
  "propertyList": [
    {
      "name": "applicationIntent", 
      "value": "ReadWrite"
    }
  ]
}

Parameters

Specifies the type of authentication to enable Dremio to connect to the source. Use ANONYMOUS when no authentication is needed and MASTER when credentials from a master database user is required.

Enum ANONYMOUS , MASTER

The SQL Server user name.

Example myUsername

The SQL Server password.

Example myPassword

The SQL Server host name.

Example localhost

The SQL Server port number. The coomonly used default port numbers for SQL Server are 1433 and 1434.

Example 1433

The database name.

Example my_database

Option to enable SSL encryption. Set to true to enable this option.

Option to show only the initial database used for connecting. Set to true to enable this option.

The number of records to fetch per request. Set to 0 to enable Dremio to automatically set the size.

Example 200

Sets the maximum number of idle connections that you want to keep.

Example 8

Sets the idle time, in seconds, before a connection is evaluated for closure.

Example 60

Option to verify the server certificate. Set to true to enable this option.

This parameter can be used when the SSL/TLS server certificate contains a distinguished name. You can leave this blank if the hostname in the certificate matches the server name.

Example localhost

An array of name/value pairs defining optional connection properties. Use a comma separated list to specify multiple name value pairs.

Example "name": "applicationIntent", "value": "ReadWrite"

Examples of Different Configurations for authenticationType

No authentication required

{
  "authenticationType": "ANONYMOUS"
}

Master credentials

{
  "authenticationType": "MASTER",
  "username": "myUsername",
  "password": "myPassword"
}

MySQL

Dremio source type: MYSQL. This type determines the config parameters that display as specified in the Source Attributes. If you are not familiar with the following parameters that are available for MySQL, see MySQL Configuration for additional information.

Config attributes for connecting to a MySQL source

"config": {
  "authenticationType": [
    "ANONYMOUS",
    "MASTER",
  ]
  "username": "myUsername",
  "password": "myPassword"
  "hostname": "localhost",
  "port": "3306",
  "netWriteTimeout": 60,
  "fetchSize": 200,
  "maxIdleConns": 8,
  "idleTimeSec": 60,
  "propertyList": [
    {
      "name": "DataSource", 
      "value": "sales2021"
    }
  ]
}

Parameters

Specifies the type of authentication to enable Dremio to connect to the source. Use ANONYMOUS when no authentication is needed and MASTER when credentials from a master database user is required.

Enum ANONYMOUS , MASTER

The MySQL user name.

Example myUsername

The MySQL password.

Example myPassword

The MySQL host name.

Example localhost

The MySQL port number. The coomonly used default port number for MySQL is 3306

Example 3306

The amount of time, in seconds, to wait for data from the server before aborting the connection.

Example 60

The number of records to fetch per request. Set to 0 to enable Dremio to automatically set the size.

Example 200

Sets the maximum number of idle connections that you want to keep.

Example 8

Sets the idle time, in seconds, before a connection is evaluated for closure.

Example 60

An array of name/value pairs defining optional connection properties. Use a comma separated list to specify multiple name value pairs.

Example "name": "DataSource", "value": "sales2021"

Examples of Different Configurations for authenticationType

No authentication required

{
  "authenticationType": "ANONYMOUS"
}

Master credentials

{
  "authenticationType": "MASTER",
  "username": "myUsername",
  "password": "myPassword"
}

Oracle

Dremio source type: ORACLE. This type determines the config parameters that display as specified in the Source Attributes. If you are not familiar with the following parameters that are available for Oracle, see Oracle Configuration for additional information.

Config attributes for connecting to an Oracle source

"config": {
  "authenticationType": [
    "ANONYMOUS",
    "MASTER"
  ]
  "username": "myUsername",
  "password": "myPassword",
  "hostname": "localhost",
  "port": "1521",
  "instance": "sales",
  "useSsl": true,
  "nativeEncryption": "ACCEPTED", 
  "useTimezoneAsRegion": true,
  "includeSynonyms": false,
  "mapDateToTimestamp": true,
  "fetchSize": 200,
  "maxIdleConns": 8,
  "idleTimeSec": 60,
  "useLdap": false,
  "bindDN": String,
  "sslServerCertDN": String,
  "propertyList": [
    {
      "name": "connect_timeout", 
      "value": "0"
    }
  ]
}

Parameters

Specifies the type of authentication to enable Dremio to connect to the source. Use `ANONYMOUS` when no authentication is needed, `MASTER` when either credentials from a master database user or a secret resource URL is required, or `KERBEROS` if Kerberos is configured.

Enum ANONYMOUS , MASTER , KERBEROS

The Oracle user name.

Example myUsername

The Oracle password.

Example myPassword

The secret resource URL.

Set to true if you are using Kerberos.

The Oracle host name.

Example localhost

The Oracle port number. The coomonly used default port numbers for Oracle are 1521 and 1830.

Example 1521

The Oracle service name.

Example sales

Option to enable SSL encryption. Set to true to enable this option.

Sets the Oracle native encryption. This parameter can only be used if useSsl is set to false.

Enum REJECTED , ACCEPTED , REQUESTED , REQUIRED

Option to use the timezone to set the connection region. Set to true to enable this option.

Option to include synonyms as datasets. Set to true to enable this option.

Option to display Oracle DATE columns as a TIMESTAMP. Set to true to enable this option.

The number of records to fetch per request. Set to 0 to enable Dremio to automatically set the size.

Example 200

Sets the maximum number of idle connections that you want to keep.

Example 8

Sets the idle time, in seconds, before a connection is evaluated for closure.

Example 60

Option to use LDAP naming services. Set to true to enable this option.

Sets the LDAP hostname resolution when the useLdap parameter is set to true. You can omit this parameter if useLdap is set to false.

Use when you have a SSL/TLS server certificate distinguished name. Otherwise, you can leave this parameter blank to disable DN match.

An array of name/value pairs defining optional connection properties. Use a comma separated list to specify multiple name value pairs.

Example "name": "connect_timeout", "value": "0"

Examples of Different Configurations for authenticationType

No authentication required

{
  "authenticationType": "ANONYMOUS"
}

Master credentials

{
  "authenticationType": "MASTER",
  "username": "myUsername",
  "password": "myPassword"
}

Secret resource URL

{
  "authenticationType": "MASTER",
  "username": "myUsername",
  "secretResourceUrl": "String"
}

Using Kerberos

{
  "credentialType": "KERBEROS",
  "username": null,
  "password": null,
  "useKerberos": true
}

PostgreSQL

Dremio source type: POSTGRES. This type determines the config parameters that display as specified in the Source Attributes. If you are not familiar with the following parameters that are available for PostgreSQL, see PostgreSQL Configuration for additional information.

Config attributes for connecting to a PostgreSQL source

"config": {
  "authenticationType": [
    "ANONYMOUS",
    "MASTER"
  ]
  "username": "myUsername",
  "password": "myPassword",
  "hostname": "localhost",
  "port": "5432",
  "databaseName": "sales_2021",
  "useSsl": false,
  "fetchSize": 200,
  "maxIdleConns": 8,
  "idleTimeSec": 60,
  "encryptionValidationMode": "CERTIFICATE_AND_HOSTNAME_VALIDATION",
  "propertyList": [
    {
      "name": "connect_timeout", 
      "value": "10&application_name=myapp"
    }
  ]
}

Parameters

Specifies the type of authentication to enable Dremio to connect to the source. Use ANONYMOUS when no authentication is needed and MASTER when either credentials from a master database user or a secret resource URL is required.

Enum ANONYMOUS , MASTER

The PostgreSQL user name.

Example myUsername

The PostgreSQL password.

Example myPassword

The secret resource URL.

The PostgreSQL host name.

Example localhost

The PostgreSQL port number. The coomonly used default port number for PostgreSQL is 5432.

Example 5432

The database name.

Example sales_2021

Option to enable SSL encryption. Set to true to enable this option.

The number of records to fetch per request. Set to 0 to enable Dremio to automatically set the size.

Example 200

Sets the maximum number of idle connections that you want to keep.

Example 8

Sets the idle time, in seconds, before a connection is evaluated for closure.

Example 60

When encryption is enabled, you can set the validation mode to one of the available presets.

Enum CERTIFICATE_AND_HOSTNAME_VALIDATION , CERTIFICATE_ONLY_VALIDATION , NO_VALIDATION

An array of name/value pairs defining optional connection properties. Use a comma separated list to specify multiple name value pairs.

Example "name": "connect_timeout", "value": "10&application_name=myapp"

Examples of Different Configurations for authenticationType

No authentication required

{
"authenticationType": "ANONYMOUS"
}

Master credentials

{
"authenticationType": "MASTER",
"username": "myUsername",
"password": "myPassword"
}

Snowflake

Dremio source type: SNOWFLAKE. This type determines the config parameters that display as specified in the Source Attributes. If you are not familiar with the following parameters that are available for Snowflake, see Configuring Snowflake as a Source for additional information.

Config attributes for connecting to a Snowflake source

"config": {
  "hostname": "https://locator_id.snowflakecomputing.com",
  "username": "myUsername",
  "password": "myPassword",
  [ "database": "myDatabase" ],
  [ "role": "PUBLIC" ],
  [ "schema": "mySchema" ],
  [ "warehouse": "myWarehouse" ]  
}

Parameters

The Snowflake host name.

Example https://locator_id.snowflakecomputing.com

The Snowflake user name.

Example myUsername

The Snowflake password.

Example myPassword

Specifies the default database to use.

Example myDatabase

Specifies the default access-control role to use. For more information, read System-Defined Roles.

Specifies the default schema to use.

Example mySchema

Specifies the warehouse that will provide resources for executing DML statements and queries.

Example myWarehouse

Was this page helpful?

Glad to hear it! Thank you for your feedback. Sorry to hear that. Thank you for your feedback.