Source Types

[info] DEPRECATED
Use the Catalog API instead. The Source API will be removed in a future release.

Dremio supports a variety of source types, each with their own custom configuration.

All editions of Dremio support the following source types:

ADL, NAS, HDFS, MAPRFS, S3, MONGO, ELASTIC, ORACLE, MYSQL, MSSQL, POSTGRES, REDSHIFT, HBASE, HIVE

The Enterprise Edition adds support for:

DB2

Azure Data Lake Store

ADL

{
  "mode": "CLIENT_KEY",
  "accountName": String,
  "clientId": String,
  "clientKeyRefreshUrl": String,
  "clientKeyPassword": String
}
Name Type Description
mode String Must be set to CLIENT_KEY
accountName String Name of the Azure Data Lake Store resource
clientId String Application ID of the registered application under Azure Active Directory
clientKeyRefreshUrl String Azure Active Directory OAuth 2.0 Token Endpoint for registered applications
clientKeyPassword String Generated password value for the registered application

NAS

NAS

{
  "path": String
}
Name Type Description
path String Path on the filesystem to use as the root for the source.

HDFS

HDFS

{
  "enableImpersonation": Boolean,
  "hostname": String,
  "port": Number,
  "rootPath": String,
  "propertyList": [
     {"name": String, "value": String},
     ...
  ]
}
Name Type Description
enableImpersonation Boolean Enable impersonation.
hostname String HDFS server host name.
port Number HDFS server port number.
rootPath String Root path for the HDFS source.
propertyList Array An array of name/value pairs.

MapR-FS

MAPRFS

{
  "clusterName": String,
  "enableImpersonation": Boolean,
  "secure": Boolean,
  "rootPath": String,
  "propertyList": [
     {"name": String, "value": String},
     ...
  ]
}
Name Type Description
clusterName String Cluster name.
enableImpersonation Boolean Enable impersonation.
secure Boolean Whether the cluster is secure or not.
rootPath String Root path for the MarR-FS source.
propertyList Array An array of name/value pairs.

Amazon S3

S3

{
  "accessKey": String,
  "accessSecret": String,
  "secure": Boolean,
  "externalBucketList": [...String],
  "propertyList": [
     {"name": String, "value": String},
     ...
  ]
}
Name Type Description
accessKey String AWS access key.
accessSecret String AWS access secret.
secure Boolean Whether to enable SSL encryption.
externalBucketList Array A list of external buckets.
propertyList Array An array of name/value pairs.

MongoDB

MONGO

{
  "username": String,
  "password": String,
  "hostList": [
     {"hostname": String, "port": Number},
     ...
  ],
  "useSsl": Boolean,
  "authenticationType": String ["ANONYMOUS", "MASTER"],
  "authDatabase": String,
  "authenticationTimeoutMillis": Number,
  "secondaryReadsOnly": Boolean,
  "subpartitionSize": Number,
  "propertyList": [
    {"name": String, "value": String},
    ...
  ]
}
Name Type Description
username String Mongo user name.
password String Mongo password.
hostList Array A list of Mongo hosts.
useSsl Boolean Force SSL connection.
authenticationType String Which authentication type to use, must be either ANONYMOUS or MASTER.
authDatabase String Authentication database.
authenticationTimeoutMillis Number Authentication time in milliseconds.
secondaryReadsOnly Boolean Read from secondaries only.
subpartitionSize Number Number of records to be read by query fragments.
propertyList Array An array of name/value pairs.

Elasticsearch

ELASTIC

{
  "username": String,
  "password": String,
  "hostList": [
     {"hostname": String, "port": Number},
     ...
  ],
  "authenticationType": String ["ANONYMOUS", "MASTER"],
  "scriptsEnabled": Boolean [optional],
  "showHiddenIndices": Boolean [optional],
  "sslEnabled": Boolean [optional],
  "showIdColumn": Boolean [optional],
  "readTimeoutMillis": Number,
  "scrollTimeoutMillis": Number,
  "usePainless": Boolean [optional],
  "useWhitelist": Boolean [optional],
  "scrollSize": Number [optional]
}
Name Type Description
username String Elasticsearch user name.
password String Elasticsearch password.
hostList Array A list of Elasticsearch hosts.
authenticationType String Which authentication type to use, must be either ANONYMOUS or MASTER.
scriptsEnabled Boolean Are scripts enabled in Elasticsearch, optional.
showHiddenIndices Boolean Whether to show hidden indices. optional.
sslEnabled Boolean Whether to use SSL connections, optional.
showIdColumn Boolean Whether to show the ID column, optional.
readTimeoutMillis Number Read timeout in milliseconds.
scrollTimeoutMillis Number Scroll timeout in milliseconds.
usePainless Boolean Whether to use the Painless scripting language when connecting to Elasticsearch 5.0+ (experimental), optional.
useWhitelist Boolean Whether to only query the specified hosts in hostList, optional.
scrollSize Number Elasticsearch scroll size, optional.

Oracle

ORACLE

{
  "username": String,
  "password": String,
  "instance": String,
  "hostname": String,
  "port": String,
  "authenticationType": String ["ANONYMOUS", "MASTER"],
  "fetchSize": Number
}
Name Type Description
username String Oracle user name.
password String Oracle password.
instance String Oracle server SID.
hostname String Oracle server host name.
port Number Oracle server port number.
authenticationType String Which authentication type to use, must be either ANONYMOUS or MASTER.
fetchSize Number Record fetch size, use 0 to have Dremio automatically decide.

MySQL

MYSQL

{
  "username": String,
  "password": String,
  "hostname": String,
  "port": String,
  "authenticationType": String ["ANONYMOUS", "MASTER"],
  "fetchSize": Number
}
Name Type Description
username String MySQL user name.
password String MySQL password.
hostname String MySQL server host name.
port Number MySQL server port number.
authenticationType String Which authentication type to use, must be either ANONYMOUS or MASTER.
fetchSize Number Record fetch size, use 0 to have Dremio automatically decide.

Microsoft SQL Server

MSSQL

{
  "username": String,
  "password": String,
  "hostname": String,
  "port": String,
  "authenticationType": String ["ANONYMOUS", "MASTER"],
  "fetchSize": Number,
  "database": String [optional],
  "showOnlyConnectiondatabase": Boolean [optional]
}
Name Type Description
username String SQL Server user name.
password String SQL Server password.
hostname String SQL Server host name.
port Number SQL Server port number.
authenticationType String Which authentication type to use, must be either ANONYMOUS or MASTER.
fetchSize Number Record fetch size, use 0 to have Dremio automatically decide.
database String Database name, optional.
showOnlyConnectiondatabase Boolean Show only the initial database used for connecting.

Postgres

POSTGRES

{
  "username": String,
  "password": String,
  "hostname": String,
  "port": String,
  "authenticationType": String ["ANONYMOUS", "MASTER"],
  "fetchSize": Number,
  "databaseName": String
}
Name Type Description
username String Postgres user name.
password String Postgres password.
hostname String Postgres host name.
port Number Postgres port number.
authenticationType String Which authentication type to use, must be either ANONYMOUS or MASTER.
fetchSize Number Record fetch size, use 0 to have Dremio automatically decide.
databaseName String Database name.

Amazon Redshift

REDSHIFT

{
  "username": String,
  "password": String,
  "authenticationType": String ["ANONYMOUS", "MASTER"],
  "fetchSize": Number,
  "connectionString": String
}
Name Type Description
username String Redshift user name.
password String Redshift password.
authenticationType String Which authentication type to use, must be either ANONYMOUS or MASTER.
fetchSize Number Record fetch size, use 0 to have Dremio automatically decide.
connectionString String Connection string.

HBase

HBASE

{
  "zkQuorum": String,
  "port": Number,
  "isSizeCalcEnabled": Boolean,
  "propertyList": [
    {"name": String, "value": String},
    ...
  ]
}
Name Type Description
zkQuorum String Zookeeper quorum, a list of comma delimited hosts.
port Number Zookeeper port.
isSizeCalcEnabled Boolean Enable region size calculation.
propertyList Array An array of name/value pairs.

Hive

HIVE

{
  "hostname": String,
  "port": String,
  "kerberosPrincipal": String,
  "enableSasl": Boolean [optional],
  "propertyList": [
    {"name": String, "value": String},
    ...
  ]
}
Name Type Description
hostname String Hive host name.
port Number Hive port number.
kerberosPrincipal String Kerberos principal.
enableSasl Boolean Enable SASL, optional.
propertyList Array An array of name/value pairs.

IBM DB2

DB2

[info] Enterprise Edition only

{
  "username": String,
  "password": String,
  "hostname": String,
  "port": String,
  "databaseName": String,
  "authenticationType": String ["ANONYMOUS", "MASTER"],
  "fetchSize": Number
}
Name Type Description
username String DB2 user name.
password String DB2 password.
hostname String DB2 host name.
port Number DB2 port number.
authenticationType String Which authentication type to use, must be either ANONYMOUS or MASTER.
fetchSize Number Record fetch size, use 0 to have Dremio automatically decide.
databaseName String Database name.

results matching ""

    No results matching ""