Using HashiCorp Vault for Secrets Management Enterprise

Dremio supports HashiCorp Vault's Kubernetes authentication method for retrieving secret references to use for connecting to data sources and listing secrets in Dremio configuration files.

After you configure Dremio to connect to Vault, you can reference a secret stored in Vault instead of using a plaintext password to configure data source connections and list secrets in Dremio configuration files. When Dremio needs a referenced Vault secret to authenticate to a data source or another service, Dremio uses Vault’s REST API to:

• Authenticate with Vault to obtain a Vault API access token.

• Get the value of the referenced secret from Vault. The reference gives Dremio the information required to make an API request to retrieve the secret. The value of the secret is only held in memory.

You can retrieve secrets from Vault's key-value version 1 (kv1) and version 2 (kv2) secrets engines. You can retrieve passwords for static roles from the Vault database secrets engine.

Supported Data Sources

The following table lists the Vault engines that Dremio supports for each data source.

Data Source	Supported Vault Engines
Amazon OpenSearch Service	kv1 and kv2
Amazon Redshift	kv1, kv2, and database
Amazon S3	kv1 and kv2
Apache Druid	kv1 and kv2
AWS Glue Data Catalog	kv1 and kv2
Azure Data Lake Storage Gen1	kv1 and kv2
Azure Storage	kv1 and kv2
Dremio-to-Dremio Connector	kv1 and kv2
Elasticsearch	kv1, kv2, and database
Google Cloud Storage	kv1 and kv2
IBM Db2	kv1, kv2, and database
Microsoft Azure Data Explorer	kv1 and kv2
Microsoft Azure Synapse Analytics	kv1 and kv2
Microsoft SQL Server	kv1, kv2, and database
MongoDB	kv1, kv2, and database
MySQL	kv1, kv2, and database
Nessie	kv1 and kv2
Oracle	kv1, kv2, and database
PostgreSQL	kv1, kv2, and database
Snowflake	kv1, kv2, and database
Teradata	kv1 and kv2

Dremio does not support the Vault integration for Hadoop Distributed File System (HDFS), Hive 2.x and 3.x, MapR File System, or network-attached storage (NAS) data sources.

Supported Configuration Files and Properties

You can use the URI for a Vault secret as the value for any of the configuration file properties listed in Encrypt Credentials. Dremio supports Vault's kv1, kv2, and database secrets engines for secrets referenced in configuration files.

Prerequisites

Ensure that you have completed the following prerequisites in Vault before you begin the integration:

• The Kubernetes authentication method does not require multi-factor authentication (MFA).

• API access tokens have a time-to-live (TTL) of at least 5 minutes.

  note

  We do not recommend configuring tokens to have an unlimited TTL.

  You may configure tokens to have a limited number of uses. However, the lower the number of uses, the more often Dremio must re-authenticate with Vault. As a result, setting the number of uses to less than 10 may degrade system performance, especially at system startup.

• The minimum credential rotation period for any Vault secret is set to 5 minutes.

• A role for Dremio to use for authentication is configured in Vault. The role must have the following access control policies:

  • The default Vault policy to allow Dremio to validate the status of its Vault API access token.

  • Read access to all Vault secrets that you want Dremio to retrieve.

Configuring Dremio to Connect to Vault

Before you can reference Vault secrets in Dremio, you must configure Dremio to connect to a single Vault. Follow these steps to properly configure Dremio:

1. Create a new JSON file for your Vault configuration to tell Dremio how to communicate and authenticate with the Vault server. To make sure the JSON file is available to all coordinator and executor pods, add it to the existing dremio-config Kubernetes ConfigMap at /opt/dremio/conf.

   JSON file for Vault configuration

   {
     "vaultUrl": "https://my-vault.com",
     "namespace": "optional/dremio/global/vault/namespace",
     "auth": {
       "kubernetes": {
         "vaultRole": "dremio-vault-role",
         "serviceAccountJwt": "file:///optional/custom/path/to/serviceAccount/jwt",
         "loginMountPath": "optional/custom/kubernetes/login/path"
       }
     }
   }

   Key	Description
   vaultUrl	The URL for Dremio to use to connect to your Vault server. Dremio sends all API requests to authenticate with Vault and retrieve Vault secrets to this URL. The vaultUrl value must use secure protocol (https) to ensure that all secrets retrieved from Vault are properly encrypted during transmission. Dremio will not connect to Vault if the vaultUrl value is not secure.
   namespace	(Optional) The global Vault namespace for Dremio to use for all authentication, secret retrieval, and API access token requests. If you include a Vault namespace in the JSON configuration file, Dremio authenticates within the namespace and can only access secrets that are available within the namespace (including sub-namespaces in the specified namespace). In API requests to Vault, Dremio sends the namespace in the X-Vault-Namespace HTTP header, so Vault’s API routing rules apply to all Vault secret URIs stored in Dremio.
   auth.kubernetes.vaultRole	The Vault role that Dremio should use for authentication. You must create the role in Vault before you configure Dremio to integrate with Vault. Once Dremio authenticates with Vault, Dremio possesses all of the access control policies associated with this role.
   auth.kubernetes.serviceAccountJwt	(Optional) The URI that specifies the filesystem location of the Kubernetes ServiceAccount JSON Web Token (JWT) that Dremio should use to authenticate with Vault. If you do not specify a serviceAccountJwt value, Dremio uses the default filesystem location for Kubernetes ServiceAccount JWTs, /var/run/secrets/kubernetes.io/serviceaccount/token.
   auth.kubernetes.loginMountPath	(Optional) The path at which the target Kubernetes authentication method was enabled. If you do not specify a loginMountPath value, Dremio sends login requests to the Vault server at /v1/auth/kubernetes: all requests use Vault's v1 API, and all authentication methods are mounted under the auth prefix. For more information, read Enabling/Disabling auth methods in the Vault documentation.

2. In your dremio.conf file, add the services.credentials.hashicorp.vault.config property and set the value to a file URI that specifies the filesystem location of your Vault JSON configuration file. For example, if your Vault JSON configuration file is located at /opt/dremio/conf/hashicorp-vault.json, the file URI is file:///opt/dremio/conf/hashicorp-vault.json.

3. Restart Dremio to propagate the configuration changes.

4. (Optional) With Dremio running, enable the services.credentials.exec.remote_lookup.enabled support key to ensure that only the master coordinator retrieves secrets from the configured Vault.

   note

   By default, both coordinator and executor pods make API requests to the configured Vault. This requires that the Kubernetes ServiceAccounts for both coordinator and executor pods can authenticate with Vault and retrieve the same secrets.

   If you enable the services.credentials.exec.remote_lookup.enabled support key, executor pods ask the master coordinator to resolve credentials for them, and the master coordinator performs the API request to Vault on behalf of the executor. With this configuration, only the master coordinator’s Kubernetes ServiceAccount needs access to Vault.

Retrieving the Secret Reference from Vault

To create a secret reference, you need to get some information about the secret from Vault. The required information varies depending on the secrets engine you use, and may include the secret engine mount path, secret path, target key, and role name. The secret reference is assembled from these components.

note

The correct format for a secret reference differs depending on the secrets engine you use and whether you will use the secret reference in the Dremio console or in a configuration file or API request. Read Formatting kv1 and kv2 Secret References and Formatting database Secret References for details and examples.

Secret Engine Mount Path (kv1, kv2, and database)

The secret engine mount path is required for kv1, kv2, and database secrets references.

In Vault, the secret engine mount path is available on the Configuration page for the secret, as shown in the image below:

Secret Path (kv1 and kv2)

The secret path is the path to the secret from the mount path of the secret engine. The secret path is required for kv1 and kv2 secrets references.

In Vault, the secret path is listed on the details page for the secret, as shown in the image below:

Target Key (kv1 and kv2)

The target key is the name used for the secret in Vault. The target key is required for kv1 and kv2 secrets references.

If the target key is not nested in a JSON object, you can retrieve it from the details page for the secret in Vault. In the example below, the JSON keys name and password are listed on the details page, and the target key is password:

When a target key is nested within another JSON object, use the key's JSON pointer to identify its location. In the example shown below, the target key, password, is nested within the credentials object. In this case, the JSON pointer credentials/password is the target key:

Role Name (database)

The role name is required for database secrets references.

The role name for a database secret is listed on the details page for the target static role:

You can also find the role name in the Roles tab for the target database secrets engine:

Formatting kv1 and kv2 Secret References

For secret references that you will use to connect to data sources in the Dremio console, the format for secret references is slightly different between kv1 and kv2 secrets:

• For the kv1 secrets engine, the format is <secret_engine_mount_path>/<secret_path>#<target_key>.
• For the kv2 secrets engine, the format is <secret_engine_mount_path>/data/<secret_path>#<target_key>.

For secret references that you will use in Dremio configuration files and API requests, add the following prefix to the secret reference you would use for the Dremio console:

• kv1 secrets engine: hashicorp-vault+kv-v1:///
• kv2 secrets engine: hashicorp-vault+kv-v2:///

Example for a Secret with No Namespace

In this example:

• The secret engine mount path is my-vault.

• The secret path is mysql/dremio-user.

• The target key, password, is listed in the following JSON object in the details page for the secret:

  {
    "password": "myPasswordValue",
    "name": "mySecret"
  }

Example kv1 secret reference for the Dremio console

my-vault/mysql/dremio-user#password

Example kv1 secret reference for a configuration file or API request

hashicorp-vault+kv-v1:///my-vault/mysql/dremio-user#password

Example kv2 secret reference for the Dremio console

my-vault/data/mysql/dremio-user#password

Example kv2 secret reference for a configuration file or API request

hashicorp-vault+kv-v2:///my-vault/data/mysql/dremio-user#password

note

Dremio does not own core-site.xml configuration files, so you must add dremio+ as a prefix to secret references before using them in core-site.xml files:

• kv1 secrets engine: dremio+hashicorp-vault+kv-v1:///my-vault/mysql/dremio-user#password
• kv2 secrets engine: dremio+hashicorp-vault+kv-v2:///my-vault/data/mysql/dremio-user#password

Example for a Secret with a Nested Target Key and No Namespace

In this example:

• The secret engine mount path is my-vault.

• The secret path is mysql/dremio-user.

• The JSON pointer to use as the target key is credentials/password, as shown in the following JSON object in the details page for the secret:

  {
    "credentials": {
      "password": "myPasswordValue"
    },
    "name": "mySecret"
  }

Example nested kv1 secret reference for the Dremio console

my-vault/mysql/dremio-user#credentials/password

Example nested kv1 secret reference for a configuration file or API request

hashicorp-vault+kv-v1:///my-vault/mysql/dremio-user#credentials/password

Example nested kv2 secret reference for the Dremio console

my-vault/data/mysql/dremio-user#credentials/password

Example nested kv2 secret reference for a configuration file or API request

hashicorp-vault+kv-v2:///my-vault/data/mysql/dremio-user#credentials/password

note

Dremio does not own core-site.xml configuration files, so you must add dremio+ as a prefix to secret references before using them in core-site.xml files:

• kv1 secrets engine: dremio+hashicorp-vault+kv-v1:///my-vault/mysql/dremio-user#credentials/password
• kv2 secrets engine: dremio+hashicorp-vault+kv-v2:///my-vault/data/mysql/dremio-user#credentials/password

Example for a Secret within a Namespace

In this example:

• The namespace is my-department.

• The secret engine mount path is my-vault.

• The secret path is mysql/dremio-user.

• The target key, password, is listed in the following JSON object in the details page for the secret:

  {
    "password": "myPasswordValue",
    "name": "mySecret"
  }

Example namespaced kv1 secret reference for the Dremio console

my-department/my-vault/mysql/dremio-user#password

Example namespaced kv1 secret reference for a configuration file or API request

hashicorp-vault+kv-v1:///my-department/my-vault/mysql/dremio-user#password

Example namespaced kv2 secret reference for the Dremio console

my-department/my-vault/data/mysql/dremio-user#password

Example namespaced kv2 secret reference for a configuration file or API request

hashicorp-vault+kv-v2:///my-department/my-vault/data/mysql/dremio-user#password

note

Dremio does not own core-site.xml configuration files, so you must add dremio+ as a prefix to secret references before using them in core-site.xml files:

• kv1 secrets engine: dremio+hashicorp-vault+kv-v1:///my-department/my-vault/mysql/dremio-user#password
• kv2 secrets engine: dremio+hashicorp-vault+kv-v2:///my-department/my-vault/data/mysql/dremio-user#password

Global namespaces are not included in secret references. To continue the example, if my-department is the global namespace, the secret references are the same as those for secrets with no namespace.

note

If your secret is in a sub-namespace of the global namespace, such as my-department/my-team you do need to include the sub-namespace my-team in the secret reference:

my-team/my-vault/data/mysql/dremio-user#password

Formatting database Secret References

For the database secrets engine, for secret references that you will use to connect to data sources, the format is <secret_engine_mount_path>/static-creds/<database_role_name>.

For secret references that you will use in Dremio configuration files and API requests, add the prefix hashicorp-vault+database+static:/// to the secret reference you would use for the Dremio console.

Example for a database Secret

In this example:

• The secret engine mount path is engineering/team-one/database.
• The secret path is my-static-role.

Example database secret reference for the Dremio console

engineering/team-one/database/static-creds/my-static-role

Example database secret reference for a configuration file or API request

hashicorp-vault+database+static:///engineering/team-one/database/static-creds/my-static-role

note

Dremio does not own core-site.xml configuration files, so you must add dremio+ as a prefix to secret references before using them in a core-site.xml file: dremio+hashicorp-vault+database+static:///engineering/team-one/database/static-creds/my-static-role.