Source Types

DEPRECATED
Use the Catalog API instead. The Source API will be removed in a future release.

Dremio supports a variety of source types, each with their own custom configuration.

All editions of Dremio support the following source types:

ADL, NAS, HDFS, MAPRFS, S3, MONGO, ELASTIC, ORACLE, MYSQL, MSSQL, POSTGRES, REDSHIFT, HBASE, HIVE

The Enterprise Edition adds support for:

DB2

Azure Data Lake Store

ADL

{
  "mode": "CLIENT_KEY",
  "accountName": String,
  "clientId": String,
  "clientKeyRefreshUrl": String,
  "clientKeyPassword": String
}
Fields
Name Type Description
mode String Must be set to CLIENT_KEY
accountName String Name of the Azure Data Lake Store resource
clientId String Application ID of the registered application under Azure Active Directory
clientKeyRefreshUrl String Azure Active Directory OAuth 2.0 Token Endpoint for registered applications
clientKeyPassword String Generated password value for the registered application

NAS

NAS

{
  "path": String
}
Fields
Name Type Description
path String Path on the filesystem to use as the root for the source.

HDFS

HDFS

{
  "enableImpersonation": Boolean,
  "hostname": String,
  "port": Number,
  "rootPath": String,
  "propertyList": [
     {"name": String, "value": String},
     ...
  ]
}
Fields
Name Type Description
enableImpersonation Boolean Enable impersonation.
hostname String HDFS server host name.
port Number HDFS server port number.
rootPath String Root path for the HDFS source.
propertyList Array An array of name/value pairs.

MapR-FS

MAPRFS

{
  "clusterName": String,
  "enableImpersonation": Boolean,
  "secure": Boolean,
  "rootPath": String,
  "propertyList": [
     {"name": String, "value": String},
     ...
  ]
}
Fields
Name Type Description
clusterName String Cluster name.
enableImpersonation Boolean Enable impersonation.
secure Boolean Whether the cluster is secure or not.
rootPath String Root path for the MarR-FS source.
propertyList Array An array of name/value pairs.

Amazon S3

S3

{
  "accessKey": String,
  "accessSecret": String,
  "secure": Boolean,
  "externalBucketList": [...String],
  "propertyList": [
     {"name": String, "value": String},
     ...
  ]
}
Fields
Name Type Description
accessKey String AWS access key.
accessSecret String AWS access secret.
secure Boolean Whether to enable SSL encryption.
externalBucketList Array A list of external buckets.
propertyList Array An array of name/value pairs.

MongoDB

MONGO

{
  "username": String,
  "password": String,
  "hostList": [
     {"hostname": String, "port": Number},
     ...
  ],
  "useSsl": Boolean,
  "authenticationType": String ["ANONYMOUS", "MASTER"],
  "authDatabase": String,
  "authenticationTimeoutMillis": Number,
  "secondaryReadsOnly": Boolean,
  "subpartitionSize": Number,
  "propertyList": [
    {"name": String, "value": String},
    ...
  ]
}
Fields
Name Type Description
username String Mongo user name.
password String Mongo password.
hostList Array A list of Mongo hosts.
useSsl Boolean Force SSL connection.
authenticationType String Which authentication type to use, must be either ANONYMOUS or MASTER.
authDatabase String Authentication database.
authenticationTimeoutMillis Number Authentication time in milliseconds.
secondaryReadsOnly Boolean Read from secondaries only.
subpartitionSize Number Number of records to be read by query fragments.
propertyList Array An array of name/value pairs.

Elasticsearch

ELASTIC

{
  "username": String,
  "password": String,
  "hostList": [
     {"hostname": String, "port": Number},
     ...
  ],
  "authenticationType": String ["ANONYMOUS", "MASTER"],
  "scriptsEnabled": Boolean [optional],
  "showHiddenIndices": Boolean [optional],
  "sslEnabled": Boolean [optional],
  "showIdColumn": Boolean [optional],
  "readTimeoutMillis": Number,
  "scrollTimeoutMillis": Number,
  "usePainless": Boolean [optional],
  "useWhitelist": Boolean [optional],
  "scrollSize": Number [optional]
}
Fields
Name Type Description
username String Elasticsearch user name.
password String Elasticsearch password.
hostList Array A list of Elasticsearch hosts.
authenticationType String Which authentication type to use, must be either ANONYMOUS or MASTER.
scriptsEnabled Boolean Are scripts enabled in Elasticsearch, optional.
showHiddenIndices Boolean Whether to show hidden indices. optional.
sslEnabled Boolean Whether to use SSL connections, optional.
showIdColumn Boolean Whether to show the ID column, optional.
readTimeoutMillis Number Read timeout in milliseconds.
scrollTimeoutMillis Number Scroll timeout in milliseconds.
usePainless Boolean Whether to use the Painless scripting language when connecting to Elasticsearch 5.0+ (experimental), optional.
useWhitelist Boolean Whether to only query the specified hosts in hostList, optional.
scrollSize Number Elasticsearch scroll size, optional.

Oracle

ORACLE

{
  "username": String,
  "password": String,
  "instance": String,
  "hostname": String,
  "port": String,
  "authenticationType": String ["ANONYMOUS", "MASTER"],
  "fetchSize": Number
}
Fields
Name Type Description
username String Oracle user name.
password String Oracle password.
instance String Oracle server SID.
hostname String Oracle server host name.
port Number Oracle server port number.
authenticationType String Which authentication type to use, must be either ANONYMOUS or MASTER.
fetchSize Number Record fetch size, use 0 to have Dremio automatically decide.

MySQL

MYSQL

{
  "username": String,
  "password": String,
  "hostname": String,
  "port": String,
  "authenticationType": String ["ANONYMOUS", "MASTER"],
  "fetchSize": Number
}
Fields
Name Type Description
username String MySQL user name.
password String MySQL password.
hostname String MySQL server host name.
port Number MySQL server port number.
authenticationType String Which authentication type to use, must be either ANONYMOUS or MASTER.
fetchSize Number Record fetch size, use 0 to have Dremio automatically decide.

Microsoft SQL Server

MSSQL

{
  "username": String,
  "password": String,
  "hostname": String,
  "port": String,
  "authenticationType": String ["ANONYMOUS", "MASTER"],
  "fetchSize": Number,
  "database": String [optional],
  "showOnlyConnectiondatabase": Boolean [optional]
}
Fields
Name Type Description
username String SQL Server user name.
password String SQL Server password.
hostname String SQL Server host name.
port Number SQL Server port number.
authenticationType String Which authentication type to use, must be either ANONYMOUS or MASTER.
fetchSize Number Record fetch size, use 0 to have Dremio automatically decide.
database String Database name, optional.
showOnlyConnectiondatabase Boolean Show only the initial database used for connecting.

Postgres

POSTGRES

{
  "username": String,
  "password": String,
  "hostname": String,
  "port": String,
  "authenticationType": String ["ANONYMOUS", "MASTER"],
  "fetchSize": Number,
  "databaseName": String
}
Fields
Name Type Description
username String Postgres user name.
password String Postgres password.
hostname String Postgres host name.
port Number Postgres port number.
authenticationType String Which authentication type to use, must be either ANONYMOUS or MASTER.
fetchSize Number Record fetch size, use 0 to have Dremio automatically decide.
databaseName String Database name.

Amazon Redshift

REDSHIFT

{
  "username": String,
  "password": String,
  "authenticationType": String ["ANONYMOUS", "MASTER"],
  "fetchSize": Number,
  "connectionString": String
}
Fields
Name Type Description
username String Redshift user name.
password String Redshift password.
authenticationType String Which authentication type to use, must be either ANONYMOUS or MASTER.
fetchSize Number Record fetch size, use 0 to have Dremio automatically decide.
connectionString String Connection string.

HBase

HBASE

{
  "zkQuorum": String,
  "port": Number,
  "isSizeCalcEnabled": Boolean,
  "propertyList": [
    {"name": String, "value": String},
    ...
  ]
}
Fields
Name Type Description
zkQuorum String Zookeeper quorum, a list of comma delimited hosts.
port Number Zookeeper port.
isSizeCalcEnabled Boolean Enable region size calculation.
propertyList Array An array of name/value pairs.

Hive

HIVE

{
  "hostname": String,
  "port": String,
  "kerberosPrincipal": String,
  "enableSasl": Boolean [optional],
  "propertyList": [
    {"name": String, "value": String},
    ...
  ]
}
Fields
Name Type Description
hostname String Hive host name.
port Number Hive port number.
kerberosPrincipal String Kerberos principal.
enableSasl Boolean Enable SASL, optional.
propertyList Array An array of name/value pairs.

IBM DB2

DB2

Only available in the Enterprise Edition.

{
  "username": String,
  "password": String,
  "hostname": String,
  "port": String,
  "databaseName": String,
  "authenticationType": String ["ANONYMOUS", "MASTER"],
  "fetchSize": Number
}
Fields
Name Type Description
username String DB2 user name.
password String DB2 password.
hostname String DB2 host name.
port Number DB2 port number.
authenticationType String Which authentication type to use, must be either ANONYMOUS or MASTER.
fetchSize Number Record fetch size, use 0 to have Dremio automatically decide.
databaseName String Database name.