Source Types
[info] DEPRECATED
Use the Catalog API instead. The Source API will be removed in a future release.
Dremio supports a variety of source types, each with their own custom configuration.
All editions of Dremio support the following source types:
ADL
, NAS
, HDFS
, MAPRFS
, S3
, MONGO
, ELASTIC
, ORACLE
, MYSQL
, MSSQL
, POSTGRES
, REDSHIFT
, HBASE
, HIVE
The Enterprise Edition adds support for:
DB2
Azure Data Lake Store
ADL
{
"mode": "CLIENT_KEY",
"accountName": String,
"clientId": String,
"clientKeyRefreshUrl": String,
"clientKeyPassword": String
}
Name | Type | Description |
---|---|---|
mode | String | Must be set to CLIENT_KEY |
accountName | String | Name of the Azure Data Lake Store resource |
clientId | String | Application ID of the registered application under Azure Active Directory |
clientKeyRefreshUrl | String | Azure Active Directory OAuth 2.0 Token Endpoint for registered applications |
clientKeyPassword | String | Generated password value for the registered application |
NAS
NAS
{
"path": String
}
Name | Type | Description |
---|---|---|
path | String | Path on the filesystem to use as the root for the source. |
HDFS
HDFS
{
"enableImpersonation": Boolean,
"hostname": String,
"port": Number,
"rootPath": String,
"propertyList": [
{"name": String, "value": String},
...
]
}
Name | Type | Description |
---|---|---|
enableImpersonation | Boolean | Enable impersonation. |
hostname | String | HDFS server host name. |
port | Number | HDFS server port number. |
rootPath | String | Root path for the HDFS source. |
propertyList | Array | An array of name/value pairs. |
MapR-FS
MAPRFS
{
"clusterName": String,
"enableImpersonation": Boolean,
"secure": Boolean,
"rootPath": String,
"propertyList": [
{"name": String, "value": String},
...
]
}
Name | Type | Description |
---|---|---|
clusterName | String | Cluster name. |
enableImpersonation | Boolean | Enable impersonation. |
secure | Boolean | Whether the cluster is secure or not. |
rootPath | String | Root path for the MarR-FS source. |
propertyList | Array | An array of name/value pairs. |
Amazon S3
S3
{
"accessKey": String,
"accessSecret": String,
"secure": Boolean,
"externalBucketList": [...String],
"propertyList": [
{"name": String, "value": String},
...
]
}
Name | Type | Description |
---|---|---|
accessKey | String | AWS access key. |
accessSecret | String | AWS access secret. |
secure | Boolean | Whether to enable SSL encryption. |
externalBucketList | Array | A list of external buckets. |
propertyList | Array | An array of name/value pairs. |
MongoDB
MONGO
{
"username": String,
"password": String,
"hostList": [
{"hostname": String, "port": Number},
...
],
"useSsl": Boolean,
"authenticationType": String ["ANONYMOUS", "MASTER"],
"authDatabase": String,
"authenticationTimeoutMillis": Number,
"secondaryReadsOnly": Boolean,
"subpartitionSize": Number,
"propertyList": [
{"name": String, "value": String},
...
]
}
Name | Type | Description |
---|---|---|
username | String | Mongo user name. |
password | String | Mongo password. |
hostList | Array | A list of Mongo hosts. |
useSsl | Boolean | Force SSL connection. |
authenticationType | String | Which authentication type to use, must be either ANONYMOUS or MASTER . |
authDatabase | String | Authentication database. |
authenticationTimeoutMillis | Number | Authentication time in milliseconds. |
secondaryReadsOnly | Boolean | Read from secondaries only. |
subpartitionSize | Number | Number of records to be read by query fragments. |
propertyList | Array | An array of name/value pairs. |
Elasticsearch
ELASTIC
{
"username": String,
"password": String,
"hostList": [
{"hostname": String, "port": Number},
...
],
"authenticationType": String ["ANONYMOUS", "MASTER"],
"scriptsEnabled": Boolean [optional],
"showHiddenIndices": Boolean [optional],
"sslEnabled": Boolean [optional],
"showIdColumn": Boolean [optional],
"readTimeoutMillis": Number,
"scrollTimeoutMillis": Number,
"usePainless": Boolean [optional],
"useWhitelist": Boolean [optional],
"scrollSize": Number [optional]
}
Name | Type | Description |
---|---|---|
username | String | Elasticsearch user name. |
password | String | Elasticsearch password. |
hostList | Array | A list of Elasticsearch hosts. |
authenticationType | String | Which authentication type to use, must be either ANONYMOUS or MASTER . |
scriptsEnabled | Boolean | Are scripts enabled in Elasticsearch, optional. |
showHiddenIndices | Boolean | Whether to show hidden indices. optional. |
sslEnabled | Boolean | Whether to use SSL connections, optional. |
showIdColumn | Boolean | Whether to show the ID column, optional. |
readTimeoutMillis | Number | Read timeout in milliseconds. |
scrollTimeoutMillis | Number | Scroll timeout in milliseconds. |
usePainless | Boolean | Whether to use the Painless scripting language when connecting to Elasticsearch 5.0+ (experimental), optional. |
useWhitelist | Boolean | Whether to only query the specified hosts in hostList, optional. |
scrollSize | Number | Elasticsearch scroll size, optional. |
Oracle
ORACLE
{
"username": String,
"password": String,
"instance": String,
"hostname": String,
"port": String,
"authenticationType": String ["ANONYMOUS", "MASTER"],
"fetchSize": Number
}
Name | Type | Description |
---|---|---|
username | String | Oracle user name. |
password | String | Oracle password. |
instance | String | Oracle server SID. |
hostname | String | Oracle server host name. |
port | Number | Oracle server port number. |
authenticationType | String | Which authentication type to use, must be either ANONYMOUS or MASTER . |
fetchSize | Number | Record fetch size, use 0 to have Dremio automatically decide. |
MySQL
MYSQL
{
"username": String,
"password": String,
"hostname": String,
"port": String,
"authenticationType": String ["ANONYMOUS", "MASTER"],
"fetchSize": Number
}
Name | Type | Description |
---|---|---|
username | String | MySQL user name. |
password | String | MySQL password. |
hostname | String | MySQL server host name. |
port | Number | MySQL server port number. |
authenticationType | String | Which authentication type to use, must be either ANONYMOUS or MASTER . |
fetchSize | Number | Record fetch size, use 0 to have Dremio automatically decide. |
Microsoft SQL Server
MSSQL
{
"username": String,
"password": String,
"hostname": String,
"port": String,
"authenticationType": String ["ANONYMOUS", "MASTER"],
"fetchSize": Number,
"database": String [optional],
"showOnlyConnectiondatabase": Boolean [optional]
}
Name | Type | Description |
---|---|---|
username | String | SQL Server user name. |
password | String | SQL Server password. |
hostname | String | SQL Server host name. |
port | Number | SQL Server port number. |
authenticationType | String | Which authentication type to use, must be either ANONYMOUS or MASTER . |
fetchSize | Number | Record fetch size, use 0 to have Dremio automatically decide. |
database | String | Database name, optional. |
showOnlyConnectiondatabase | Boolean | Show only the initial database used for connecting. |
Postgres
POSTGRES
{
"username": String,
"password": String,
"hostname": String,
"port": String,
"authenticationType": String ["ANONYMOUS", "MASTER"],
"fetchSize": Number,
"databaseName": String
}
Name | Type | Description |
---|---|---|
username | String | Postgres user name. |
password | String | Postgres password. |
hostname | String | Postgres host name. |
port | Number | Postgres port number. |
authenticationType | String | Which authentication type to use, must be either ANONYMOUS or MASTER . |
fetchSize | Number | Record fetch size, use 0 to have Dremio automatically decide. |
databaseName | String | Database name. |
Amazon Redshift
REDSHIFT
{
"username": String,
"password": String,
"authenticationType": String ["ANONYMOUS", "MASTER"],
"fetchSize": Number,
"connectionString": String
}
Name | Type | Description |
---|---|---|
username | String | Redshift user name. |
password | String | Redshift password. |
authenticationType | String | Which authentication type to use, must be either ANONYMOUS or MASTER . |
fetchSize | Number | Record fetch size, use 0 to have Dremio automatically decide. |
connectionString | String | Connection string. |
HBase
HBASE
{
"zkQuorum": String,
"port": Number,
"isSizeCalcEnabled": Boolean,
"propertyList": [
{"name": String, "value": String},
...
]
}
Name | Type | Description |
---|---|---|
zkQuorum | String | Zookeeper quorum, a list of comma delimited hosts. |
port | Number | Zookeeper port. |
isSizeCalcEnabled | Boolean | Enable region size calculation. |
propertyList | Array | An array of name/value pairs. |
Hive
HIVE
{
"hostname": String,
"port": String,
"kerberosPrincipal": String,
"enableSasl": Boolean [optional],
"propertyList": [
{"name": String, "value": String},
...
]
}
Name | Type | Description |
---|---|---|
hostname | String | Hive host name. |
port | Number | Hive port number. |
kerberosPrincipal | String | Kerberos principal. |
enableSasl | Boolean | Enable SASL, optional. |
propertyList | Array | An array of name/value pairs. |
IBM DB2
DB2
[info] Enterprise Edition only
{
"username": String,
"password": String,
"hostname": String,
"port": String,
"databaseName": String,
"authenticationType": String ["ANONYMOUS", "MASTER"],
"fetchSize": Number
}
Name | Type | Description |
---|---|---|
username | String | DB2 user name. |
password | String | DB2 password. |
hostname | String | DB2 host name. |
port | Number | DB2 port number. |
authenticationType | String | Which authentication type to use, must be either ANONYMOUS or MASTER . |
fetchSize | Number | Record fetch size, use 0 to have Dremio automatically decide. |
databaseName | String | Database name. |