On this page

    Prerequisites in AWS and Permissions Required by Dremio

    Before you sign up, you need to ensure that you meet a few prerequisites.

    As part of signing up, you grant a number of permissions to Dremio on your Amazon Virtual Private Cloud (Amazon VPC).

    Prerequisites

    • An AWS account and region that you want to use with Dremio Cloud. The regions that Dremio Cloud supports are listed here.

    • An Amazon Virtual Private Cloud (Amazon VPC) and subnets in which to give Dremio Cloud permissions on storage, compute, and network resources.
      Guidelines for selecting subnets:

      • Ensure that subnet IDs are unique across all of the availability zones within an Amazon VPC.
      • Ensure that each subnet that you specify belongs to a separate availability zone; for example, if you specify subnet A and subnet B, they cannot both be in availability zone C, but must be in separate availability zones.
      • Specify only private subnets or only public subnets; mixing private and public subnets is not supported.
    • Your AWS user account must have permission to run a CloudFormation Template that creates the following items:

      • A security group, which acts as a virtual firewall to control the traffic that is allowed to and from the resources that the security group is associated with. The security group ensures that only traffic from Dremio Cloud reaches the resources that you have allocated for your Dremio Cloud organization. In addition to the security group, the CloudFormation template creates an outbound rule that allows compute engines to connect to Dremio Cloud’s control plane by using TLS.
      • An S3 bucket for storing various types of project data, including:
        • The data for reflections that are created in the project
        • File uploads to home spaces
        • The default path for new tables that are used for data and manifests for datasets
        • All of the tables that store records of events and other historical data
        • Dremio encrypts the S3 bucket used for the project store, and you can select the encryption type in the CloudFormation template during the final step of the automatic sign-up process (Step 3 of 3 - Configure AWS resources). The default encryption mechanism is SSE-S3, or you can select SSE-KMS (AWS Managed Keys) or SSE-KMS (Customer Managed Keys). If you select SSE-KMS (Customer Managed Keys), you must provide the ARN for an existing KMS Customer Managed Key, and ensure that the AWS user account has access to the specified key.
      • An AWS Identity and Access Management (IAM) role or IAM user that is granted the permissions that are described in Policy Template to Grant Access to the Project Store and Policy Template for Enabling Dremio Cloud to Manage Engines
        • An IAM role is an IAM identity that you can create in your account that has specific permissions. In this case, the IAM roles are granted permissions on the resources that you specify for your Dremio Cloud organization, and these roles are assigned to Dremio Cloud.
        • An IAM user is an entity that you create in AWS to represent the person or application that uses it to interact with AWS. A user in AWS consists of a name and credentials. In this case, Dremio Cloud is given the access key ID and secret access key as credentials for connecting to your VPC to access the resources that you give it permission to use.
    • Outbound connectivity from your Amazon VPC and subnets is required to allow query engines to communicate with Dremio Cloud. Engines establish a connection with the Dremio Cloud control plane using port 443 (HTTPS) outbound to the internet. No open ports are required in your Amazon VPC, and neither subnets for incoming connections nor engines require public IP addresses.

      To verify outbound connectivity from your subnets, run the following command from an EC2 instance within each subnet:

      Example Request
      curl -v https://gw.dremio.cloud
      

    Policy Template to Grant Access to the Project Store

    The following policy template is the minimum policy requirement to allow read and write access to the project store. It grants Dremio Cloud permissions, through IAM roles or IAM users, for storing metadata and views for the project in an S3 bucket in your Amazon VPC. The permissions are described in comments in the template. Replace BUCKET-NAME with the name of the S3 bucket you want to use as the Dremio Cloud project store:

    Policy Template
    {
      "Version": "2012-10-17",
      "Statement": [
        # Allow Dremio to enumerate S3 buckets within the account.
        {
          "Effect": "Allow",
          "Action": [
            "s3:ListBucket",
            "s3:ListAllMyBuckets"
          ],
          "Resource": "*"
        },
        # Allow Dremio read and write access to the Project Store bucket used to store housekeeping information such as metadata and reflections.
        {
          "Effect": "Allow",
          "Action": [
            "s3:DeleteObject",
            "s3:GetObject",
            "s3:PutObject"
          ],
          "Resource": [
            "arn:aws:s3:::BUCKET-NAME/*"
          ]
        },
        # Allow Dremio to determine the region, list content and add tags on the Project Store bucket.
        {
          "Effect": "Allow",
          "Action": [
            "s3:GetBucketLocation",
            "s3:ListBucket",
            "s3:PutBucketTagging"
          ],
          "Resource": [
            "arn:aws:s3:::BUCKET-NAME"
          ]
        },
        # Allow Dremio read access to sample datasets used to get users started easily on the platform without connecting their own data.
        {
          "Effect": "Allow",
          "Action": [
            "s3:ListBucket",
            "s3:GetBucketLocation",
            "s3:GetObject"
          ],
          "Resource": [
            "arn:aws:s3:::ap-southwest-1.examples.dremio.com",
            "arn:aws:s3:::eu-west-1.examples.dremio.com",
            "arn:aws:s3:::us-east-1.examples.dremio.com",
            "arn:aws:s3:::us-west-1.examples.dremio.com",
            "arn:aws:s3:::us-west-2.examples.dremio.com"
          ]
        }
      ]
    }
    

    Policy Template for Enabling Dremio Cloud to Manage Engines

    The following policy enables Dremio Cloud to create and manage engines in your AWS VPC. The permissions are described in comments in the template:

    Policy Template
    {
      "Version": "2012-10-17",
      "Statement": [
        # Allow Dremio to terminate instances with the "dremio_managed" tag.
        {
          "Effect": "Allow",
          "Action": "ec2:TerminateInstances",
          "Resource": "arn:aws:ec2:*:*:instance/*",
          "Condition": {
            "StringEquals": {
              "ec2:ResourceTag/dremio_managed": "true"
            }
          }
        },
        # Require the "dremio_managed" tag for instances/volumes when creating instances.
        {
          "Effect": "Allow",
          "Action": "ec2:RunInstances",
          "Resource": [
            "arn:aws:ec2:*:*:volume/*",
            "arn:aws:ec2:*:*:instance/*"
          ],
          "Condition": {
            "StringEquals": {
              "aws:RequestTag/dremio_managed": "true"
            }
          }
        },
        # Allow creating instances without the "dremio_managed" tag on resources other than instances/volumes.
        {
          "Effect": "Allow",
          "Action": "ec2:RunInstances",
          "Resource": [
            "arn:aws:ec2:*:*:launch-template/*",
            "arn:aws:ec2:*:*:fleet/*",
            "arn:aws:ec2:*::image/*",
            "arn:aws:ec2:*:*:network-interface/*",
            "arn:aws:ec2:*:*:security-group/*",
            "arn:aws:ec2:*:*:subnet/*",
            "arn:aws:ec2:*:*:placement-group/*"
          ]
        },
        # Allow Dremio to create tags on instances/volumes only upon the initial creation of an instance.
        {
          "Effect": "Allow",
          "Action": "ec2:CreateTags",
          "Resource": [
            "arn:aws:ec2:*:*:instance/*",
            "arn:aws:ec2:*:*:volume/*"
          ],
          "Condition": {
            "StringEquals": {
              "ec2:CreateAction": "RunInstances"
            }
          }
        },
        # Allow Dremio to create tags on placement groups (PG) upon the initial creation of a PG.
        {
          "Effect": "Allow",
          "Action": "ec2:CreateTags",
          "Resource": "arn:aws:ec2:*:*:placement-group/*",
          "Condition": {
            "StringEquals": {
              "ec2:CreateAction": "CreatePlacementGroup"
            }
          }
        },
        # Allow Dremio to create tags on a launch template (LT) upon the initial creation of a LT.
        {
          "Effect": "Allow",
          "Action": "ec2:CreateTags",
          "Resource": "arn:aws:ec2:*:*:launch-template/*",
          "Condition": {
            "StringEquals": {
              "ec2:CreateAction": "CreateLaunchTemplate"
            }
          }
        },
        # Allow Dremio to create tags on a fleet upon the initial creation of the fleet.
        {
          "Effect": "Allow",
          "Action": "ec2:CreateTags",
          "Resource": "arn:aws:ec2:*:*:fleet/*",
          "Condition": {
            "StringEquals": {
              "ec2:CreateAction": "CreateFleet"
            }
          }
        },
        # Allow Dremio to create fleet only when including the "dremio_managed" tag.
        {
          "Effect": "Allow",
          "Action": "ec2:CreateFleet",
          "Resource": "arn:aws:ec2:*:*:fleet/*",
          "Condition": {
            "StringEquals": {
              "aws:RequestTag/dremio_managed": "true"
            }
          }
        },
        # Allow Dremio to create fleet with other resources without the "dremio_managed" tag.
        {
          "Effect": "Allow",
          "Action": "ec2:CreateFleet",
          "Resource": [
            "arn:aws:ec2:*:*:instance/*",
            "arn:aws:ec2:*:*:image/*",
            "arn:aws:ec2:*:*:launch-template/*",
            "arn:aws:ec2:*:*:network-interface/*",
            "arn:aws:ec2:*:*:placement-group/*",
            "arn:aws:ec2:*:*:security-group/*",
            "arn:aws:ec2:*:*:subnet/*"
          ]
        },
        # Only allow Dremio to delete fleets with the "dremio_managed" tag.
        {
          "Effect": "Allow",
          "Action": "ec2:DeleteFleets",
          "Resource": "arn:aws:ec2:*:*:fleet/*",
          "Condition": {
            "StringEquals": {
              "ec2:ResourceTag/dremio_managed": "true"
            }
          }
        },
        # Allow Dremio to create a launch template.
        {
          "Effect": "Allow",
          "Action": "ec2:CreateLaunchTemplate",
          "Resource": "arn:aws:ec2:*:*:launch-template/*"
        },
        # Only allow Dremio to delete launch templates with the "dremio_managed" tag.
        {
          "Effect": "Allow",
          "Action": "ec2:DeleteLaunchTemplate",
          "Resource": "arn:aws:ec2:*:*:launch-template/*",
          "Condition": {
            "StringEquals": {
              "ec2:ResourceTag/dremio_managed": "true"
            }
          }
        },
        # Allow Dremio to describe fleets with the "dremio_managed" tag.
        {
          "Effect": "Allow",
          "Action": "ec2:DescribeFleets",
          "Resource": "arn:aws:ec2:*:*:fleet/*",
          "Condition": {
            "StringEquals": {
              "ec2:ResourceTag/dremio_managed": "true"
            }
          }
        },
        # Only allow Dremio to delete placement groups with the "dremio_managed" tag.
        {
          "Effect": "Allow",
          "Action": "ec2:DeletePlacementGroup",
          "Resource": "arn:aws:ec2:*:*:placement-group/*",
          "Condition": {
            "StringEquals": {
              "ec2:ResourceTag/dremio_managed": "true"
            }
          }
        },
        # Allow Dremio to create a placement group.
        {
          "Effect": "Allow",
          "Action": "ec2:CreatePlacementGroup",
          "Resource": "arn:aws:ec2:*:*:placement-group/*"
        },
        # Allow Dremio to enumerate resources in the account.
        {
          "Effect": "Allow",
          "Action": [
            "ec2:DescribeImages",
            "ec2:DescribeLaunchTemplateVersions",
            "ec2:DescribeLaunchTemplates",
            "ec2:DescribeVpcs",
            "ec2:DescribeSubnets",
            "ec2:DescribeTags",
            "ec2:DescribeAvailabilityZones",
            "ec2:DescribeInstances",
            "ec2:DescribeInstanceStatus",
            "ec2:DescribeNetworkInterfaces",
            "ec2:DescribeNetworkInterfaceAttribute",
            "ec2:DescribePlacementGroups",
            "ec2:DescribeSecurityGroups",
            "ec2:DescribeVpcEndpoints",
            "ec2:DescribeVolumes"
          ],
          "Resource": "*"
        },
        # This section is necessary only if you are using a storage role.
        {
          "Effect": "Allow",
          "Action": [
            "iam:PassRole",
            "sts:AssumeRole"
          ],
          "Resource": [
            "<Storage Role ARN>"
          ]
        }
      ]
    }