Using AWS Secrets Manager for Secrets Management

Use Dremio's AWS Secrets Manager integration to reference a secret rather than providing a password or other sensitive information when you connect to a data source or configure a Dremio configuration file.

note

Dremio must be deployed on AWS to use the AWS Secrets Manager integration for secrets management.

After you complete the prerequisites and create the secret in the AWS Secrets Manager console or AWS command line interface (CLI) or API, Dremio can use the secret reference you provide to retrieve the secret's value at runtime and authenticate to a data source or another service.

Supported Data Sources

Dremio supports AWS Secrets Manager secrets for all data source types except Hadoop Distributed File System (HDFS), Hive 2.x and 3.x, MapR File System, and network-attached storage (NAS).

Supported Configuration Files and Properties

You can use an AWS Secrets Manager secret as the value for any of the configuration file properties listed in Encrypt Credentials.

Prerequisites

To authenticate with AWS Secrets Manager, Dremio uses the EC2 instance profile of the EC2 instance on which Dremio is running. For this reason, an EC2 instance profile must be attached to each EC2 instance on which Dremio coordinators and executors are running.

For Dremio to retrieve the values of secrets at runtime, you must grant the EC2 instance profile the ability to perform the following actions on the target secrets in AWS Secrets Manager:

• secretsmanager:GetSecretValue
• secretsmanager:DescribeSecret

Make sure to grant these permissions to Dremio’s EC2 instance profile only on the secrets that you intend for Dremio to read. Read the AWS Secrets Manager instructions for configuring access control policies, attaching these policies to an identity (the associated IAM role for the EC2 instance profile), and attaching these policies to an AWS Secrets Manager secret to ensure that Dremio can read only the desired secrets.

When Dremio is running within an Amazon EKS cluster, you must grant the secretsmanager:GetSecretValue and secretsmanager:DescribeSecret permissions to the EKS node group’s IAM role, not to a ServiceAccount-based IAM role or an IAM role that is associated with an EKS pod identity.

Secrets Created in the AWS Secrets Manager Console

Dremio supports the following types of secrets created in the AWS Secrets Manager console:

• Credentials for Amazon RDS databases.

• Credentials for Amazon Redshift cluster.

• Credentials for other databases.

• Secrets like generic key/value pairs and raw secret string options.

  • For generic key/value pairs and raw secret string options, you must map the target secret to the password key (which is case-sensitive) or the secret’s value must be the plaintext secret. If the secret’s value is a JSON object, Dremio ignores values that are mapped to keys other than password. Even if the JSON object contains other key-value pairs, Dremio only reads the value mapped to the password key.

    The following image shows a generic AWS Secrets Manager secret that correctly maps the password key to the target password value:

    

    The following image shows a generic AWS Secrets Manager secret that correctly sets the secret's value to the plaintext password:

    

Secrets Created with the AWS CLI or API

Dremio supports AWS Secrets Manager secrets created with the AWS command line interface (CLI) or API. Secrets created with the AWS CLI must be secret strings specified using the --secret-string argument. The value of the secret must be either a JSON object in which the target secret value is mapped to the top-level password key (which is case-sensitive) or the plaintext secret value.

note

Dremio does not support using binary data for secret values.

If the secret value is a JSON object, Dremio ignores values that are mapped to any other keys besides password. Even if the JSON object contains other key-value pairs, Dremio only reads the value mapped to the password key.

Example secret in a JSON object

{
  "password": "plaintext-password-value-here"
}

Example secret in a JSON object that contains other key-value pairs

{
  "password": "plaintext-password-value-here",
  "username": "my-dremio-user",
  "host": "localhost"
}

Example plaintext secret

plaintext-password-value-here

Retrieving the Secret Reference from AWS Secrets Manager

The secret reference for an AWS Secrets Manager secret is the secret’s Amazon Resource Name (ARN). The ARN is available on the secret's details page in the AWS Secrets Manager console:

Using the Secret Reference when Connecting to Data Sources

When you configure a new data source or edit the settings for an existing data source, enter the ARN for the AWS Secrets Manager secret directly into the corresponding password or secret key field in the Dremio console. If you configure a data source using the Dremio API, provide the ARN as the value for the corresponding parameter in your request body.

Using the Secret Reference in Dremio Configuration Files

In Dremio configuration files, use the ARN for the AWS Secrets Manager secret in place of a plaintext secret. You can use the secret's ARN as the value for any of the configuration file properties listed in Encrypt Credentials.

note

You must add dremio+ as a prefix to secret references before using them in core-site.xml files. For example:

dremio+arn:aws:secretsmanager:us-west-2:01234567890:secret:mysql-test/dremio-user-adKre