> > >

Source Type Configuraton

This topic specifies the source types that Dremio supports. A variety of source types are supported, each with their own custom configuration.

The following source types are available:

Source	Source Type
Amazon Elasticsearch Service	AMAZONELASTIC
Amazon Redshift	REDSHIFT
Amazon S3	S3
Azure Data Lake Storage Gen1	ADLS
Azure Storage	AZURE_STORAGE
Elasticsearch	ELASTIC
HDFS	HDFS
Hive	HIVE
MapR-FS	MAPRFS
Microsoft SQL Server	MSSQL
MongoDB	MONGO
MySQL	MYSQL
NAS	NAS
Oracle	ORACLE
PostgreSQL	POSTGRES

Amazon Elasticsearch Service

AMAZONELASTIC

{
    "hostname":            String,
    "port":                Number,
    "authenticationType":  String [ACCESS_KEY, EC2_METADATA, NONE],
    "accessKey":           String,
    "accessSecret":        String,
    "overwriteRegion":     Boolean,
    "regionName":          String,
    "scriptsEnabled":      Boolean,
    "showHiddenIndices":   Boolean,
    "showIdColumn":        Boolean,
    "readTimeoutMillis":   Number,
    "scrollTimeoutMillis": Number,
    "usePainless":         Boolean,
    "scrollSize":          Number,
    "allowPushdownOnNormalizedOrAnalyzedFields": Boolean,
    "warnOnRowCountMismatch":      Boolean,
    "encryptionValidationMode":    String [
        CERTIFICATE_AND_HOSTNAME_VALIDATION, CERTIFICATE_ONLY_VALIDATION, NO_VALIDATION
        ]
}

Amazon Redshift

REDSHIFT

{
  "username": String,
  "password": String,
  "authenticationType": String ["ANONYMOUS", "MASTER"],
  "fetchSize": Number,
  "connectionString": String
}

Name	Type	Description
username	String	Redshift user name.
password	String	Redshift password.
authenticationType	String	Which authentication type to use, must be either `ANONYMOUS` or `MASTER`.
fetchSize	Number	Record fetch size, use `0` to have Dremio automatically decide.
connectionString	String	Connection string.

Amazon S3

S3

{
  "accessKey": String,
  "accessSecret": String,
  "secure": Boolean,
  "externalBucketList": [...String],
  "propertyList": [
     {"name": String, "value": String},
     ...
  ]
}

Name	Type	Description
accessKey	String	AWS access key.
accessSecret	String	AWS access secret.
secure	Boolean	Whether to enable SSL encryption.
externalBucketList	Array	A list of external buckets.
propertyList	Array	An array of name/value pairs.

Azure Data Lake Store (Gen1)

ADL

{
  "mode": "CLIENT_KEY",
  "accountName": String,
  "clientId": String,
  "clientKeyRefreshUrl": String,
  "clientKeyPassword": String
}

Name	Type	Description
mode	String	Must be set to `CLIENT_KEY`
accountName	String	Name of the Azure Data Lake Store resource
clientId	String	Application ID of the registered application under Azure Active Directory
clientKeyRefreshUrl	String	Azure Active Directory OAuth 2.0 Token Endpoint for registered applications
clientKeyPassword	String	Generated password value for the registered application

Azure Storage

AZURE_STORAGE

Shared Access Key

Azure Storage V2 or V1 source with Shared Access Key:

"config": {
  "accountKind": String,
  "accountName": String,
  "accessKey": String,
  "enableSSL" : Boolean,
  "rootPath": String,
  "containers" : [... String]
  "propertyList": [
     {"name": String, "value": String},
     ...
  ]
}

Name	Type	Description
accountKind	String	Set to either “STORAGE_V1” or “STORAGE_V2”.
accountName	String	Name of the Azure storage account.
accessKey	String	Shared access key of the Azure storage account.
enableSSL	Boolean	Whether to enable SSL encryption.
rootPath	String	Set to “/” to view all accessible containers. Alternatively, the path to a container can be specified instead of “/” to set it as the root, e.g. “/container-name“.
containers	Array	A list of containers to be accessed. When specified, only containers listed will be available for access.
propertyList	Array	An array of name/value pairs.

OAuth 2.0 Authentication

Azure Storage V2 or V1 source with OAuth 2.0 authentication:

"config": {
  "accountKind": String,
  "accountName": String,
  "credentialsType": String,
  "clientId": String,
  "tokenEndpoint": String,
  "clientSecret": String
  "enableSSL" : Boolean,
  "rootPath": String,
  "containers" : [... String]
  "propertyList": [
     {"name": String, "value": String},
     ...
  ]
}

Name	Type	Description
accountKind	String	Set to either “STORAGE_V1” or “STORAGE_V2”.
accountName	String	Name of the Azure storage account.
credentialsType	String	Must be set to “AZURE_ACTIVE_DIRECTORY” when using OAuth 2.0.
clientId	String	Application (Client) ID.
tokenEndpoint	String	OAuth 2.0 token endpoint (v1.0).
clientSecret	String	Secret key generated in Azure Active Directory application.
enableSSL	Boolean	Must be set to true when using OAuth 2.0.
rootPath	String	Set to “/” to view all accessible containers. Alternatively, the path to a container can be specified instead of “/” to set it as the root, e.g. “/container-name“.
containers	Array	A list of containers to be accessed. When specified, only containers listed will be available for access.
propertyList	Array	An array of name/value pairs.

Elasticsearch

ELASTIC

{
  "username": String,
  "password": String,
  "hostList": [
     {"hostname": String, "port": Number},
     ...
  ],
  "authenticationType": String ["ANONYMOUS", "MASTER"],
  "scriptsEnabled": Boolean [optional],
  "showHiddenIndices": Boolean [optional],
  "sslEnabled": Boolean [optional],
  "showIdColumn": Boolean [optional],
  "readTimeoutMillis": Number,
  "scrollTimeoutMillis": Number,
  "usePainless": Boolean [optional],
  "useWhitelist": Boolean [optional],
  "scrollSize": Number [optional]
}

Name	Type	Description
username	String	Elasticsearch user name.
password	String	Elasticsearch password.
hostList	Array	A list of Elasticsearch hosts.
authenticationType	String	Which authentication type to use, must be either `ANONYMOUS` or `MASTER`.
scriptsEnabled	Boolean	Are scripts enabled in Elasticsearch, optional.
showHiddenIndices	Boolean	Whether to show hidden indices. optional.
sslEnabled	Boolean	Whether to use SSL connections, optional.
showIdColumn	Boolean	Whether to show the ID column, optional.
readTimeoutMillis	Number	Read timeout in milliseconds.
scrollTimeoutMillis	Number	Scroll timeout in milliseconds.
usePainless	Boolean	Whether to use the Painless scripting language when connecting to Elasticsearch 5.0+ (experimental), optional.
useWhitelist	Boolean	Whether to only query the specified hosts in hostList, optional.
scrollSize	Number	Elasticsearch scroll size, optional.

HDFS

HDFS

{
  "enableImpersonation": Boolean,
  "hostname": String,
  "port": Number,
  "rootPath": String,
  "propertyList": [
     {"name": String, "value": String},
     ...
  ]
}

Name	Type	Description
enableImpersonation	Boolean	Enable impersonation.
hostname	String	HDFS server host name.
port	Number	HDFS server port number.
rootPath	String	Root path for the HDFS source.
propertyList	Array	An array of name/value pairs.

Hive

HIVE

{
  "hostname": String,
  "port": String,
  "kerberosPrincipal": String,
  "enableSasl": Boolean [optional],
  "propertyList": [
    {"name": String, "value": String},
    ...
  ]
}

Name	Type	Description
hostname	String	Hive host name.
port	Number	Hive port number.
kerberosPrincipal	String	Kerberos principal.
enableSasl	Boolean	Enable SASL, optional.
propertyList	Array	An array of name/value pairs.

MapR-FS

MAPRFS

{
  "clusterName": String,
  "enableImpersonation": Boolean,
  "secure": Boolean,
  "rootPath": String,
  "propertyList": [
     {"name": String, "value": String},
     ...
  ]
}

Name	Type	Description
clusterName	String	Cluster name.
enableImpersonation	Boolean	Enable impersonation.
secure	Boolean	Whether the cluster is secure or not.
rootPath	String	Root path for the MarR-FS source.
propertyList	Array	An array of name/value pairs.

Microsoft SQL Server

MSSQL

{
  "username": String,
  "password": String,
  "hostname": String,
  "port": String,
  "authenticationType": String ["ANONYMOUS", "MASTER"],
  "fetchSize": Number,
  "database": String [optional],
  "showOnlyConnectiondatabase": Boolean [optional]
}

Name	Type	Description
username	String	SQL Server user name.
password	String	SQL Server password.
hostname	String	SQL Server host name.
port	Number	SQL Server port number.
authenticationType	String	Which authentication type to use, must be either `ANONYMOUS` or `MASTER`.
fetchSize	Number	Record fetch size, use `0` to have Dremio automatically decide.
database	String	Database name, optional.
showOnlyConnectiondatabase	Boolean	Show only the initial database used for connecting.

MongoDB

MONGO

{
  "username": String,
  "password": String,
  "hostList": [
     {"hostname": String, "port": Number},
     ...
  ],
  "useSsl": Boolean,
  "authenticationType": String ["ANONYMOUS", "MASTER"],
  "authDatabase": String,
  "authenticationTimeoutMillis": Number,
  "secondaryReadsOnly": Boolean,
  "subpartitionSize": Number,
  "propertyList": [
    {"name": String, "value": String},
    ...
  ]
}

Name	Type	Description
username	String	Mongo user name.
password	String	Mongo password.
hostList	Array	A list of Mongo hosts.
useSsl	Boolean	Force SSL connection.
authenticationType	String	Which authentication type to use, must be either `ANONYMOUS` or `MASTER`.
authDatabase	String	Authentication database.
authenticationTimeoutMillis	Number	Authentication time in milliseconds.
secondaryReadsOnly	Boolean	Read from secondaries only.
subpartitionSize	Number	Number of records to be read by query fragments.
propertyList	Array	An array of name/value pairs.

MySQL

MYSQL

{
  "username": String,
  "password": String,
  "hostname": String,
  "port": String,
  "authenticationType": String ["ANONYMOUS", "MASTER"],
  "fetchSize": Number
}

Name	Type	Description
username	String	MySQL user name.
password	String	MySQL password.
hostname	String	MySQL server host name.
port	Number	MySQL server port number.
authenticationType	String	Which authentication type to use, must be either `ANONYMOUS` or `MASTER`.
fetchSize	Number	Record fetch size, use `0` to have Dremio automatically decide.

NAS

NAS

{
  "path": String
}

Name	Type	Description
path	String	Path on the filesystem to use as the root for the source.

Oracle

ORACLE

{
  "username": String,
  "password": String,
  "instance": String,
  "hostname": String,
  "port": String,
  "authenticationType": String ["ANONYMOUS", "MASTER"],
  "fetchSize": Number
}

Name	Type	Description
username	String	Oracle user name.
password	String	Oracle password.
instance	String	Oracle server SID.
hostname	String	Oracle server host name.
port	Number	Oracle server port number.
authenticationType	String	Which authentication type to use, must be either `ANONYMOUS` or `MASTER`.
fetchSize	Number	Record fetch size, use `0` to have Dremio automatically decide.

PostgreSQL

POSTGRES

{
  "username": String,
  "password": String,
  "hostname": String,
  "port": String,
  "authenticationType": String ["ANONYMOUS", "MASTER"],
  "fetchSize": Number,
  "databaseName": String
}

Name	Type	Description
username	String	Postgres user name.
password	String	Postgres password.
hostname	String	Postgres host name.
port	Number	Postgres port number.
authenticationType	String	Which authentication type to use, must be either `ANONYMOUS` or `MASTER`.
fetchSize	Number	Record fetch size, use `0` to have Dremio automatically decide.
databaseName	String	Database name.

Deprecated Sources

The following data source types are deprecated.

HBase

As of Dremio 3.2, HBase is deprecated for existing configurations and not supported for new.

HBASE

{
  "zkQuorum": String,
  "port": Number,
  "isSizeCalcEnabled": Boolean,
  "propertyList": [
    {"name": String, "value": String},
    ...
  ]
}

Name	Type	Description
zkQuorum	String	Zookeeper quorum, a list of comma delimited hosts.
port	Number	Zookeeper port.
isSizeCalcEnabled	Boolean	Enable region size calculation.
propertyList	Array	An array of name/value pairs.

IBM DB2

As of Dremio 3.2, IBM DB2 is deprecated for existing configurations and not supported for new.

DB2

Enterprise Edition only

{
  "username": String,
  "password": String,
  "hostname": String,
  "port": String,
  "databaseName": String,
  "authenticationType": String ["ANONYMOUS", "MASTER"],
  "fetchSize": Number
}

Name	Type	Description
username	String	DB2 user name.
password	String	DB2 password.
hostname	String	DB2 host name.
port	Number	DB2 port number.
authenticationType	String	Which authentication type to use, must be either `ANONYMOUS` or `MASTER`.
fetchSize	Number	Record fetch size, use `0` to have Dremio automatically decide.
databaseName	String	Database name.