Using Azure Key Vault for Secrets Management

Use Dremio's Azure Key Vault integration to reference a secret rather than providing a password or other sensitive information when you connect to a data source or configure a Dremio configuration file.

note

Dremio must be deployed on Azure to use the Azure Key Vault integration for secrets management.

After you complete the prerequisites, you can create secrets in Azure Key Vault and use them in Dremio. Dremio uses the secret references you provide to retrieve secret values at runtime and authenticate to data sources and other services.

Supported Data Sources

Dremio supports Azure Key Vault secrets for all data source types except Hadoop Distributed File System (HDFS), Hive 2.x and 3.x, MapR File System, and network-attached storage (NAS).

Supported Configuration Files and Properties

You can use an Azure Key Vault secret as the value for any of the configuration file properties listed in Encrypt Credentials.

Prerequisites

Dremio uses Azure managed identities to connect to Azure Key Vault. Create a user-managed identity for Dremio or select an existing user-managed identity previously created for Dremio. Note the Client ID of your managed identity.

Example: The Client ID of the managed identity is shown on the right as d91a5b36-189d-4137-bf1f-7c0a36ef11b0.

To enable Dremio to connect to Azure Key Vault using your managed identity, complete the following steps:

AKS

When deploying Dremio on a Kubernetes cluster in the Azure Kubernetes Service (AKS):

1. Assign the managed identity to the AKS cluster's Virtual Machine Scale Set (VMSS).

2. Provide access to Azure Key Vault using Azure RBAC access control or an Azure access policy. When using Azure RBAC, grant access to the Azure Key Value Secret User role.

3. Pull the latest version of the Dremio Kubernetes Helm Chart.

4. Edit dremio-cloud-tools/dremio_v2/values.yaml in your local copy of the Dremio Helm chart. Remove the leading # to expose extraEnvs in the coordinator and executor sections and add the AZURE_MANAGED_IDENTITY_CLIENT_ID variable with the value of your managed identity Client ID.

   Example updates in values.yaml

   coordinator:
      ...
      # Extra Environment Variables
      # Uncomment below to add extra environment variables for the coordinator.
      extraEnvs:
         - name: AZURE_MANAGED_IDENTITY_CLIENT_ID
           value: d91a5b36-189d-4137-bf1f-7c0a36ef11b0
   
   executor:
      ...
      # Extra Environment Variables
      # Uncomment below to add extra environment variables for the executors.
      extraEnvs:
         - name: AZURE_MANAGED_IDENTITY_CLIENT_ID
           value: d91a5b36-189d-4137-bf1f-7c0a36ef11b0

5. Deploy new Dremio containers using your updated Helm chart.

Azure VMs

When deploying Dremio on Azure virtual machines (VMs):

1. Assign the managed identity to each coordinator and executor VM.

2. Provide access to Azure Key Vault using Azure RBAC access control or an Azure access policy. When using Azure RBAC, grant access to the Azure Key Value Secret User role.

3. Edit the Dremio dremio-env file on the coordinator and each executor to add the AZURE_MANAGED_IDENTITY_CLIENT_ID environment variable with the Client ID of your managed identity. Export this variable definition from dremio-env.

   Example updates in dremio-env

   export AZURE_MANAGED_IDENTITY_CLIENT_ID="d91a5b36-189d-4137-bf1f-7c0a36ef11b0"

4. Restart Dremio to use your updated dremio-env.

Retrieving the Secret Reference from Azure Key Vault

The secret reference for an Azure Key Vault secret is the secret identifier URL, without the secret version number. The secret identifier is available on the secret's details page in the Azure Key Vault console.

Example: The Azure Key Vault secret value for use in Dremio is https://sourcecreds.vault.azure.net/secrets/azurestorage.

Using the Secret Reference when Connecting to Data Sources

When you configure a new data source or edit the settings for an existing data source, enter the partial secret identifier URL for the Azure Key Vault secret directly into the corresponding password or secret key field in the Dremio console. If you configure a data source using the Dremio API, provide the partial secret identifier as the value for the corresponding parameter in your request body.

Using the Secret Reference in Dremio Configuration Files

In Dremio configuration files, use the partial secret identifier URL for the Azure Key Vault secret in place of a plaintext secret. You can use the partial secret identifier URL as the value for any of the configuration file properties listed in Encrypt Credentials.

note

You must add dremio+ as a prefix to secret references before using them in core-site.xml files. For example:

dremio+https://sourcecreds.vault.azure.net/secrets/azurestorage