Prerequisites for Configuring Cloud Resources

Before running queries with Dremio Sonar, you must first provision the required resources in your cloud provider. You can then configure these cloud resources in the next step when adding your Sonar project.

Prerequisites

For the configuration, you will need to address the following prerequisites:

Use an AWS account
Choose a supported region that you want to use
Create or use an existing Amazon Virtual Private Cloud (VPC) and subnets
Establish outbound connectivity from your VPC and subnets to allow query engines to communicate with Dremio Cloud
Grant permissions to Dremio Cloud for storage, compute, and network resources

Connecting Your AWS Account

Currently, Dremio Cloud only supports AWS, so you will need an AWS account.

If you don't have access to an AWS account with the required permissions, you can sign up for an AWS Free Tier account at https://aws.amazon.com/free/.

Selecting a VPC and Subnets

You can use an existing VPC and subnets, although if you don't have a VPC that meets the networking requirements, then you will need to create one. For steps, see Create a VPC and Subnets.

See the following guidelines for selecting subnets:

Specify only private subnets or only public subnets. Mixing private and public subnets is not supported.
Ensure that each subnet that you specify belongs to a separate availability zone. For example, if you specify subnet A and subnet B, they cannot both be in availability zone C, but they must be in separate availability zones.
Ensure that subnet IDs are unique across all of the availability zones within a VPC.

Establishing Outbound Connectivity

Outbound connectivity is required to allow query engines to communicate with Dremio Cloud. Engines establish a connection with the Dremio Cloud control plane using port 443 (HTTPS). No other open ports are required in your VPC.

To establish this connection, you can use an internet gateway with a public IP address, NAT gateway, or AWS PrivateLink. If your VPC has internet connectivity, you can securely connect to the Dremio Cloud control plane via the internet gateway or NAT gateway. However, we recommend using PrivateLink as it provides secure connectivity to the Dremio Cloud control plane and also improves the overall security posture as it does not require the VPC to have internet connectivity. In addition, we provide a CloudFormation template to simplify the provisioning of the PrivateLink.

Verify Connectivity

Before getting started with Dremio Sonar, verify outbound connectivity from your subnets by running the following command from an EC2 instance in each subnet:

Command to verify connectivity

curl -v https://gw.dremio.cloud

Granting Permissions

You must also grant Dremio some permissions on your VPC. You can choose either to:

Grant them automatically by launching the CloudFormation template (CFT) from your AWS user account. The CFT is the recommended method, because the CFT will create the required resources for you. For a breakdown of the resources that will be created and the permissions that will be granted, see the annotated CloudFormation template.
Grant them manually by following the steps listed on Configuring Cloud Resources Manually if you prefer to create the required resources and Sonar project manually. For a breakdown of the resources that you will need to create, see Required Resources.

To use either method, you must have permissions to create the following required resources and grant the required access:

Storage creates an S3 bucket with read and write permissions to use as the project store. The project store contains all project-specific data, such as metadata and reflections.
Compute Access creates an AWS Identity and Access Management (IAM) role or user that will create and manage compute resources (Dremio engines). An IAM role or IAM user is grant access to the project store and enables Dremio Cloud to manage engines.
Network creates a security group with an outbound rule that allows connectivity from Dremio engines to the Dremio control plane via TLS.

Required Resources

If you choose to configure the cloud resources manually, you will need to create the following required resources. Otherwise, the CFT will create the required resources for you.

S3 Bucket

An encrypted S3 bucket is used for the project store that stores various types of project data, including:

The data for reflections that are created in the project
The default path for new tables that are used for data and manifests for datasets
All of the tables that store records of events and other historical data

Security Group

A security group acts as a virtual firewall to control the traffic that is allowed to and from your resources, ensuring that only traffic from Dremio Cloud reaches the resources that you have allocated for your Dremio Cloud organization.

Outbound Rule

An outbound rule allows EC2 instances to connect to Dremio’s control plane by using TLS. For example, if the VPC for your organization is running in AWS, Dremio’s control plane deploys compute engines as AWS EC2 instances within your VPC.

IAM Role or IAM User

An IAM role is an IAM identity that you can create in your account that has specific permissions. In this case, the IAM roles are granted permissions on the resources that you specify for your Dremio Cloud organization, and these roles are assigned to Dremio Cloud.

An IAM user is an entity that you create in AWS to represent the person or application that uses it to interact with AWS. A user in AWS consists of a name and credentials. In this case, Dremio Cloud is given the access key ID and secret access key as credentials for connecting to your VPC to access the resources that you give it permission to use.

Policy Template to Grant Access to the Project Store

The following policy template is the minimum policy requirement to allow read and write access to the project store. It grants Dremio Cloud permissions, through IAM roles or IAM users, for storing metadata and views for the project in an S3 bucket in your Amazon VPC. The permissions are described in comments in the template. Replace BUCKET-NAME with the name of the S3 bucket you want to use as the Dremio Cloud project store:

Template for the Policy JSON

{
  "Version": "2012-10-17",
  "Statement": [
    # Allow Dremio to enumerate S3 buckets within the account.
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListAllMyBuckets"
      ],
      "Resource": "arn:aws:s3:::*"
    },
    # Allow Dremio R/W access to the Project Store bucket used to store housekeeping information such as metadata and reflections.
    {
      "Effect": "Allow",
      "Action": [
        "s3:DeleteObject",
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:s3:::BUCKET-NAME/*"
      ]
    },
    # Allow Dremio to determine the region, list content and add tags on the Project Store bucket.
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetBucketLocation",
        "s3:ListBucket",
        "s3:PutBucketTagging"
      ],
      "Resource": [
        "arn:aws:s3:::BUCKET-NAME"
      ]
    },
    # Allow Dremio read access to sample datasets used to get users started easily on the platform without connecting their own data.
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::ap-southwest-1.examples.dremio.com",
        "arn:aws:s3:::eu-west-1.examples.dremio.com",
        "arn:aws:s3:::us-east-1.examples.dremio.com",
        "arn:aws:s3:::us-west-1.examples.dremio.com",
        "arn:aws:s3:::us-west-2.examples.dremio.com"
      ]
    }
  ]
}  

Policy Template for Enabling Dremio Cloud to Manage Engines

The following policy enables Dremio Cloud to create and manage engines in your VPC. The permissions are described in comments in the template:

Template for the Policy JSON

{
  "Version": "2012-10-17",
  "Statement": [
    # Allow Dremio to terminate instances with the "dremio_managed" tag.
    {
      "Effect": "Allow",
      "Action": "ec2:TerminateInstances",
      "Resource": "arn:aws:ec2:*:*:instance/*",
      "Condition": {
        "StringEquals": {
          "ec2:ResourceTag/dremio_managed": "true"
        }
      }
    },
    # Require the "dremio_managed" tag for instances/volumes when creating instances.
    {
      "Effect": "Allow",
      "Action": "ec2:RunInstances",
      "Resource": [
        "arn:aws:ec2:*:*:volume/*",
        "arn:aws:ec2:*:*:instance/*"
      ],
      "Condition": {
        "StringEquals": {
          "aws:RequestTag/dremio_managed": "true"
        }
      }
    },
    # Allow creating instances without the "dremio_managed" tag on resources other than instances/volumes.
    {
      "Effect": "Allow",
      "Action": "ec2:RunInstances",
      "Resource": [
        "arn:aws:ec2:*:*:launch-template/*",
        "arn:aws:ec2:*:*:fleet/*",
        "arn:aws:ec2:*::image/*",
        "arn:aws:ec2:*:*:network-interface/*",
        "arn:aws:ec2:*:*:security-group/*",
        "arn:aws:ec2:*:*:subnet/*",
        "arn:aws:ec2:*:*:placement-group/*"
      ]
    },
    # Allow Dremio to create tags on instances/volumes only upon the initial creation of an instance.
    {
      "Effect": "Allow",
      "Action": "ec2:CreateTags",
      "Resource": [
        "arn:aws:ec2:*:*:instance/*",
        "arn:aws:ec2:*:*:volume/*"
      ],
      "Condition": {
        "StringEquals": {
          "ec2:CreateAction": "RunInstances"
        }
      }
    },
    # Allow Dremio to create tags on placement groups (PG) upon the initial creation of a PG.
    {
      "Effect": "Allow",
      "Action": "ec2:CreateTags",
      "Resource": "arn:aws:ec2:*:*:placement-group/*",
      "Condition": {
        "StringEquals": {
          "ec2:CreateAction": "CreatePlacementGroup"
        }
      }
    },
    # Allow Dremio to create tags on a launch template (LT) upon the initial creation of a LT.
    {
      "Effect": "Allow",
      "Action": "ec2:CreateTags",
      "Resource": "arn:aws:ec2:*:*:launch-template/*",
      "Condition": {
        "StringEquals": {
          "ec2:CreateAction": "CreateLaunchTemplate"
        }
      }
    },
    # Allow Dremio to create tags on a fleet upon the initial creation of the fleet.
    {
      "Effect": "Allow",
      "Action": "ec2:CreateTags",
      "Resource": "arn:aws:ec2:*:*:fleet/*",
      "Condition": {
        "StringEquals": {
          "ec2:CreateAction": "CreateFleet"
        }
      }
    },
    # Allow Dremio to create fleet only when including the "dremio_managed" tag.
    {
      "Effect": "Allow",
      "Action": "ec2:CreateFleet",
      "Resource": "arn:aws:ec2:*:*:fleet/*",
      "Condition": {
        "StringEquals": {
          "aws:RequestTag/dremio_managed": "true"
        }
      }
    },
    # Allow Dremio to create fleet with other resources without the "dremio_managed" tag.
    {
      "Effect": "Allow",
      "Action": "ec2:CreateFleet",
      "Resource": [
        "arn:aws:ec2:*:*:instance/*",
        "arn:aws:ec2:*:*:image/*",
        "arn:aws:ec2:*:*:launch-template/*",
        "arn:aws:ec2:*:*:network-interface/*",
        "arn:aws:ec2:*:*:placement-group/*",
        "arn:aws:ec2:*:*:security-group/*",
        "arn:aws:ec2:*:*:subnet/*"
      ]
    },
    # Only allow Dremio to delete fleets with the "dremio_managed" tag.
    {
      "Effect": "Allow",
      "Action": "ec2:DeleteFleets",
      "Resource": "arn:aws:ec2:*:*:fleet/*",
      "Condition": {
        "StringEquals": {
          "ec2:ResourceTag/dremio_managed": "true"
        }
      }
    },
    # Allow Dremio to create a launch template.
    {
      "Effect": "Allow",
      "Action": "ec2:CreateLaunchTemplate",
      "Resource": "arn:aws:ec2:*:*:launch-template/*"
    },
    # Only allow Dremio to delete launch templates with the "dremio_managed" tag.
    {
      "Effect": "Allow",
      "Action": "ec2:DeleteLaunchTemplate",
      "Resource": "arn:aws:ec2:*:*:launch-template/*",
      "Condition": {
        "StringEquals": {
          "ec2:ResourceTag/dremio_managed": "true"
        }
      }
    },
    # Allow Dremio to describe fleets with the "dremio_managed" tag.
    {
      "Effect": "Allow",
      "Action": "ec2:DescribeFleets",
      "Resource": "arn:aws:ec2:*:*:fleet/*",
      "Condition": {
        "StringEquals": {
          "ec2:ResourceTag/dremio_managed": "true"
        }
      }
    },
    # Only allow Dremio to delete placement groups with the "dremio_managed" tag.
    {
      "Effect": "Allow",
      "Action": "ec2:DeletePlacementGroup",
      "Resource": "arn:aws:ec2:*:*:placement-group/*",
      "Condition": {
        "StringEquals": {
          "ec2:ResourceTag/dremio_managed": "true"
        }
      }
    },
    # Allow Dremio to create a placement group.
    {
      "Effect": "Allow",
      "Action": "ec2:CreatePlacementGroup",
      "Resource": "arn:aws:ec2:*:*:placement-group/*"
    },
    # Allow Dremio to enumerate resources in the account.
    {
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeImages",
        "ec2:DescribeLaunchTemplateVersions",
        "ec2:DescribeLaunchTemplates",
        "ec2:DescribeVpcs",
        "ec2:DescribeSubnets",
        "ec2:DescribeTags",
        "ec2:DescribeAvailabilityZones",
        "ec2:DescribeInstances",
        "ec2:DescribeInstanceStatus",
        "ec2:DescribeNetworkInterfaces",
        "ec2:DescribeNetworkInterfaceAttribute",
        "ec2:DescribePlacementGroups",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeVpcEndpoints",
        "ec2:DescribeVolumes"
      ],
      "Resource": "*"
    },
    # This section appears only if you chose to create a cross-account IAM role in the previous step.
    {
      "Effect": "Allow",
      "Action": [
        "iam:PassRole",
        "sts:AssumeRole"
      ],
      "Resource": [
        "<Role ARN from Step 1: Configure Storage Settings>"
      ]
    }
  ]
}

Wrap-up and Next Steps

To configure your cloud resources, see Getting Started with Dremio Sonar.
For additional information, see AWS Resources.

Prerequisites for Configuring Cloud Resources

Prerequisites​

Connecting Your AWS Account​

Selecting a VPC and Subnets​

Establishing Outbound Connectivity​

Verify Connectivity​

Granting Permissions​

Required Resources​

S3 Bucket​

Security Group​

Outbound Rule​

IAM Role or IAM User​

Policy Template to Grant Access to the Project Store​

Policy Template for Enabling Dremio Cloud to Manage Engines​

Wrap-up and Next Steps​