Dremio Cloud
Dremio Software
 
On this page

    Source Types

    Deprecated:

    Use the Catalog API instead. The Source API will be removed in a future release.

    Dremio supports a variety of source types, each with their own custom configuration.

    All editions of Dremio support the following source types:

    ADL, NAS, HDFS, MAPRFS, S3, MONGO, ELASTIC, ORACLE, MYSQL, MSSQL, POSTGRES, REDSHIFT, HIVE

    Azure Data Lake Store

    Azure Data Lake Store source 
    {
  "mode": "CLIENT_KEY",
  "accountName": String,
  "clientId": String,
  "clientKeyRefreshUrl": String,
  "clientKeyPassword": String
}
    Name Type Description
    mode String Must be set to CLIENT_KEY
    accountName String Name of the Azure Data Lake Store resource
    clientId String Application ID of the registered application under Azure Active Directory
    clientKeyRefreshUrl String Azure Active Directory OAuth 2.0 Token Endpoint for registered applications
    clientKeyPassword String Generated password value for the registered application

    NAS

    NAS source 
    {
  "path": String
}
    Name Type Description
    path String Path on the filesystem to use as the root for the source.

    HDFS

    HDFS source 
    {
  "enableImpersonation": Boolean,
  "hostname": String,
  "port": Number,
  "rootPath": String,
  "propertyList": [
     {"name": String, "value": String},
     ...
  ]
}
    Name Type Description
    enableImpersonation Boolean Enable impersonation.
    hostname String HDFS server host name.
    port Number HDFS server port number.
    rootPath String Root path for the HDFS source.
    propertyList Array An array of name/value pairs.

    MapR-FS

    MapR-FS source 
    {
  "clusterName": String,
  "enableImpersonation": Boolean,
  "secure": Boolean,
  "rootPath": String,
  "propertyList": [
     {"name": String, "value": String},
     ...
  ]
}
    Name Type Description
    clusterName String Cluster name.
    enableImpersonation Boolean Enable impersonation.
    secure Boolean Whether the cluster is secure or not.
    rootPath String Root path for the MarR-FS source.
    propertyList Array An array of name/value pairs.

    Amazon S3

    S3 source 
    {
  "accessKey": String,
  "accessSecret": String,
  "secure": Boolean,
  "externalBucketList": [...String],
  "propertyList": [
     {"name": String, "value": String},
     ...
  ]
}
    Name Type Description
    accessKey String AWS access key.
    accessSecret String AWS access secret.
    secure Boolean Whether to enable SSL encryption.
    externalBucketList Array A list of external buckets.
    propertyList Array An array of name/value pairs.

    MongoDB

    MongoDB source 
    {
  "username": String,
  "password": String,
  "hostList": [
     {"hostname": String, "port": Number},
     ...
  ],
  "useSsl": Boolean,
  "authenticationType": String ["ANONYMOUS", "MASTER"],
  "authDatabase": String,
  "authenticationTimeoutMillis": Number,
  "secondaryReadsOnly": Boolean,
  "subpartitionSize": Number,
  "propertyList": [
    {"name": String, "value": String},
    ...
  ]
}
    Name Type Description
    username String Mongo user name.
    password String Mongo password.
    hostList Array A list of Mongo hosts.
    useSsl Boolean Force SSL connection.
    authenticationType String Which authentication type to use, must be either ANONYMOUS or MASTER.
    authDatabase String Authentication database.
    authenticationTimeoutMillis Number Authentication time in milliseconds.
    secondaryReadsOnly Boolean Read from secondaries only.
    subpartitionSize Number Number of records to be read by query fragments.
    propertyList Array An array of name/value pairs.

    Elasticsearch

    Elasticsearch source 
    {
  "username": String,
  "password": String,
  "hostList": [
     {"hostname": String, "port": Number},
     ...
  ],
  "authenticationType": String ["ANONYMOUS", "MASTER"],
  "scriptsEnabled": Boolean [optional],
  "showHiddenIndices": Boolean [optional],
  "sslEnabled": Boolean [optional],
  "showIdColumn": Boolean [optional],
  "readTimeoutMillis": Number,
  "scrollTimeoutMillis": Number,
  "usePainless": Boolean [optional],
  "useWhitelist": Boolean [optional],
  "scrollSize": Number [optional]
}
    Name Type Description
    username String Elasticsearch user name.
    password String Elasticsearch password.
    hostList Array A list of Elasticsearch hosts.
    authenticationType String Which authentication type to use, must be either ANONYMOUS or MASTER.
    scriptsEnabled Boolean Are scripts enabled in Elasticsearch, optional.
    showHiddenIndices Boolean Whether to show hidden indices. optional.
    sslEnabled Boolean Whether to use SSL connections, optional.
    showIdColumn Boolean Whether to show the ID column, optional.
    readTimeoutMillis Number Read timeout in milliseconds.
    scrollTimeoutMillis Number Scroll timeout in milliseconds.
    usePainless Boolean Whether to use the Painless scripting language when connecting to Elasticsearch 5.0+ (experimental), optional.
    useWhitelist Boolean Whether to only query the specified hosts in hostList, optional.
    scrollSize Number Elasticsearch scroll size, optional.

    Oracle

    Oracle source 
    {
  "username": String,
  "password": String,
  "instance": String,
  "hostname": String,
  "port": String,
  "authenticationType": String ["ANONYMOUS", "MASTER"],
  "fetchSize": Number
}
    Name Type Description
    username String Oracle user name.
    password String Oracle password.
    instance String Oracle server SID.
    hostname String Oracle server host name.
    port Number Oracle server port number.
    authenticationType String Which authentication type to use, must be either ANONYMOUS or MASTER.
    fetchSize Number Record fetch size, use 0 to have Dremio automatically decide.

    MySQL

    MySQL source 
    {
  "username": String,
  "password": String,
  "hostname": String,
  "port": String,
  "authenticationType": String ["ANONYMOUS", "MASTER"],
  "fetchSize": Number
}
    Name Type Description
    username String MySQL user name.
    password String MySQL password.
    hostname String MySQL server host name.
    port Number MySQL server port number.
    authenticationType String Which authentication type to use, must be either ANONYMOUS or MASTER.
    fetchSize Number Record fetch size, use 0 to have Dremio automatically decide.

    Microsoft SQL Server

    Microsoft SQL Server source 
    {
  "username": String,
  "password": String,
  "hostname": String,
  "port": String,
  "authenticationType": String ["ANONYMOUS", "MASTER"],
  "fetchSize": Number,
  "database": String [optional],
  "showOnlyConnectiondatabase": Boolean [optional]
}
    Name Type Description
    username String SQL Server user name.
    password String SQL Server password.
    hostname String SQL Server host name.
    port Number SQL Server port number.
    authenticationType String Which authentication type to use, must be either ANONYMOUS or MASTER.
    fetchSize Number Record fetch size, use 0 to have Dremio automatically decide.
    database String Database name, optional.
    showOnlyConnectiondatabase Boolean Show only the initial database used for connecting.

    Postgres

    PostgreSQL source 
    {
  "username": String,
  "password": String,
  "hostname": String,
  "port": String,
  "authenticationType": String ["ANONYMOUS", "MASTER"],
  "fetchSize": Number,
  "databaseName": String
}
    Name Type Description
    username String Postgres user name.
    password String Postgres password.
    hostname String Postgres host name.
    port Number Postgres port number.
    authenticationType String Which authentication type to use, must be either ANONYMOUS or MASTER.
    fetchSize Number Record fetch size, use 0 to have Dremio automatically decide.
    databaseName String Database name.

    Amazon Redshift

    Redshift source 
    {
  "username": String,
  "password": String,
  "authenticationType": String ["ANONYMOUS", "MASTER"],
  "fetchSize": Number,
  "connectionString": String
}
    Name Type Description
    username String Redshift user name.
    password String Redshift password.
    authenticationType String Which authentication type to use, must be either ANONYMOUS or MASTER.
    fetchSize Number Record fetch size, use 0 to have Dremio automatically decide.
    connectionString String Connection string.

    Hive

    Hive source 
    {
  "hostname": String,
  "port": String,
  "kerberosPrincipal": String,
  "enableSasl": Boolean [optional],
  "propertyList": [
    {"name": String, "value": String},
    ...
  ]
}
    Name Type Description
    hostname String Hive host name.
    port Number Hive port number.
    kerberosPrincipal String Kerberos principal.
    enableSasl Boolean Enable SASL, optional.
    propertyList Array An array of name/value pairs.

    Deprecated Source Types

    The following data source types are deprecated and no longer supported.

    • HBase

    • IBM DB2