On this page

    dbt preview

    dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.

    You can use Dremio’s dbt connector dbt-dremio to transform data that is in data sources that are connected to a Dremio project.

    Prerequisites

    • Download the dbt-dremio package from https://github.com/dremio/dbt-dremio.

    • Ensure that Python 3.9.x or later is installed.

    • Ensure that you are using Dremio Software version 22.0 or later.

    • Enable these support keys in your Dremio cluster:

      • dremio.iceberg.enabled
      • dremio.iceberg.ctas.enabled
      • dremio.execution.support_unlimited_splits

      See Support Keys in the Dremio documentation for the steps.

    • If you want to use TLS to secure the connection between dbt and Dremio Software, configure full wire encryption in your Dremio cluster. For instructions, see Configuring Wire Encryption.

    Installing

    Install this package from PyPi by running this command:

    pip install dbt-dremio
    

    note:

    dbt-dremio includes dbt core v1.3.0. If you have an earlier version of dbt core installed, dbt-dremio updates it to v1.3.0.

    Initializing a dbt Project

    1. Run the command dbt init <project_name>.
    2. Select dremio as the database to use.
    3. Select one of these options to generate a profile for your project:
      • software_with_username_password for working with a Dremio Software cluster and authenticating to the cluster with a username and a password
      • software_with_pat for working with a Dremio Software cluster and authenticating to the cluster with a personal access token

    Next, configure the profile for your dbt project.

    Profiles

    When you initialize a dbt project, you create one of these three profiles. You must configure it before trying to connect to Dremio Cloud or Dremio Software.

    For descriptions of the configurations in these profiles, see Configurations.

    Profile for Dremio Software with Username/Password Authentication

    Example Profile
    [project name]:
      outputs:
        dev:
          password: b9JtkIgI3uup9gGxxK
          port: 9047
          software_host: 192.0.2.0
          object_storage_source: Samples
          object_storage_path: "samples.dremio.com"."Dremio University"
          dremio_space: Space1
          dremio_space_folder: Folder1.Folder2
          threads: 1
          type: dremio
          use_ssl: true
          user: userName
      target: dev
    

    Profile for Dremio Software with Authentication Through a Personal Access Token

    Example Profile
    [project name]:
      outputs:
        dev:
          pat: A1BCDrE2FwgH3IJkLM4NoPqrsT5uV6WXyza7I8bcDEFgJ9hIj0Kl1MNOPq2Rstu
          port: 9047
          software_host: 192.0.2.0
          object_storage_source: Samples
          object_storage_path: "samples.dremio.com"."Dremio University"
          dremio_space: Space1
          dremio_space_folder: Folder1.Folder2
          threads: 1
          type: dremio
          use_ssl: true
          user: userName
      target: dev
    

    Configurations

    ConfigurationRequired?Default ValueDescription
    passwordYes, if you are not using the pat configuration.NoneThe password of the account to use when logging into the Dremio cluster.
    patYes, if you are not using the user and password configurations.NoneThe personal access token to use for authenticating to Dremio. See Personal Access Tokens for instructions about obtaining a token. The use of a personal access token takes precedence if values for the three configurations user, password and pat are specified.
    portYes9047Port for Dremio Software cluster API endpoints.
    software_hostYesNoneThe hostname or IP address of the coordinator node of the Dremio cluster.
    object_storage_sourceNo$scratchThe name of the filesystem in which to create tables, materialized views, tests, and other objects. The dbt alias is datalake.
    This name corresponds to the name of a source in the Object Storage section of the Datasets page in Dremio:
    object_storage_pathNono_schemaThe path in the filesystem in which to create objects. The default is the root level of the filesystem. The dbt alias is root_path. Nested folders in the path are separated with periods.
    This value corresponds to the path in this location in the Datasets page in Dremio: 'samples.dremio.com'.'Dremio University'
    dremio_spaceNo@<username>The value of the Dremio space in which to create views. The dbt alias is database.
    This value corresponds to the name in this location in the Spaces section of the Datasets page in Dremio: Spaces1
    dremio_space_folderNono_schemaThe folder in the Dremio space in which to create views. The default is the top level in the space. The dbt alias is schema. Nested folders are separated with periods.
    This value corresponds to the path in this location in the Datasets page in Dremio: Folder1.Folder2
    threadsYes1The number of threads the dbt project runs on.
    typeYesdremioAuto-populated when creating a Dremio project. Do not change this value.
    use_sslYestrueAcceptable values are true and false. If the value is set to true, ensure that full wire encryption is configured in your Dremio cluster. See Prerequisites for Dremio Software.
    userYesNoneThe username of the account to use when logging into the Dremio cluster.