On this page

    Connecting Your AWS Account to Dremio Cloud Manually

    On the Cloud Connect page of the sign-up process, you can choose to manually create AWS resources and grant Dremio Cloud access to them. Click Manual on that page and then follow the steps listed below.

    Step 1: Configure Storage Settings

    In the Storage Settings section of the Cloud Connect page, configure the S3 bucket for Dremio Cloud to use as the metadata store for your project.

    When you configure storage settings, you grant Dremio Cloud read and write permissions to the S3 bucket. Those permissions are defined in the policy JSON. See the comments in the template for the policy JSON below.

    Template for the Policy JSON
    {
      "Version": "2012-10-17",
      "Statement": [
        # Allow Dremio to enumerate S3 buckets within the account.
        {
          "Effect": "Allow",
          "Action": [
            "s3:ListBucket",
            "s3:ListAllMyBuckets"
          ],
          "Resource": "*"
        },
        # Allow Dremio R/W access to the Project Store bucket used to store housekeeping information such as metadata and reflections.
        {
          "Effect": "Allow",
          "Action": [
            "s3:DeleteObject",
            "s3:GetObject",
            "s3:PutObject"
          ],
          "Resource": [
            "arn:aws:s3:::BUCKET-NAME/*"
          ]
        },
        # Allow Dremio to determine the region, list content and add tags on the Project Store bucket.
        {
          "Effect": "Allow",
          "Action": [
            "s3:GetBucketLocation",
            "s3:ListBucket",
            "s3:PutBucketTagging"
          ],
          "Resource": [
            "arn:aws:s3:::BUCKET-NAME"
          ]
        },
        # Allow Dremio read access to sample datasets used to get users started easily on the platform without connecting their own data.
        {
          "Effect": "Allow",
          "Action": [
            "s3:ListBucket",
            "s3:GetBucketLocation",
            "s3:GetObject"
          ],
          "Resource": [
            "arn:aws:s3:::ap-southwest-1.examples.dremio.com",
            "arn:aws:s3:::eu-west-1.examples.dremio.com",
            "arn:aws:s3:::us-east-1.examples.dremio.com",
            "arn:aws:s3:::us-west-1.examples.dremio.com",
            "arn:aws:s3:::us-west-2.examples.dremio.com"
          ]
        }
      ]
    }
    

    To configure storage settings:

    1. Create an Amazon S3 bucket.

      For the steps, see AWS: Create a Dremio Cloud Project Store.

    2. In the Project Store field on the Cloud Connect page, paste the path of the Amazon S3 bucket. Doing so sets the path in the Resource section of the policy JSON. The rest of the policy JSON sets the minimum policy requirement to allow Dremio Cloud read and write access to the project store.

    3. Copy the content of the Policy JSON text box and paste it in a location that you can access later in this procedure.

    4. To grant Dremio Cloud read and write permissions to the project store, follow either of these two sets of steps:

      • To grant the permissions by means of an IAM user:

        a. In the Project Data Credentials field, select Access Key.

        b. Create an IAM user with an access key.

        c. In the AWS Access Key ID field, paste the Access Key ID from the .csv file you downloaded when you created the user.

        d. In the AWS Secret Access Key field, paste the AWS Secret Access Key from the .csv file you downloaded when you created the user.

      • To grant the permissions by means of a cross-account IAM role:

        a. In the Project Data Credentials field, select IAM Role.

        b. Copy this JSON snippet and paste it into a location that you can access for the rest of this procedure.

            {
              "Effect": "Allow",
              "Principal": {
                "AWS": "arn:aws:iam::<trust account ID>:root"
              },
              "Action": "sts:AssumeRole",
              "Condition": {
                "StringEquals": {
                  "sts:ExternalId": "<external ID>"
                }
              }
            }
        

        c. Copy the trust account ID and the external ID and paste them into the JSON that you copied in the previous step.

        d. Create a cross-account IAM role in the AWS Console.

        e. In the Role ARN field, paste the cross-account role ARN that you copied when you created the role.

        f. In the Instance Profile ARN field, paste the instance profile ARN that you copied when you created the role.

    Step 2: Configure Compute Settings

    In the Compute Settings section of the Cloud Connect page, specify the credentials for Dremio Cloud to use to create and manage compute engines in your Amazon VPC.

    When you configure compute settings, you grant Dremio Cloud permissions to create and manage compute instances for Dremio engines. As with the storage settings, you can grant the permissions by means of an IAM user or an IAM role.

    note:

    You can use the same IAM user that you created for granting access to storage settings by adding an additional JSON policy to the existing IAM user. For steps on how to attach new permission policies to an existing IAM user, see setting up AWS permissions.

    Permissions are granted only on resources that are tagged as dremio_managed. Those permissions are defined in the policy JSON. See the comments in the template for the policy JSON below.

    Template for the Policy JSON
    {
      "Version": "2012-10-17",
      "Statement": [
        # Allow Dremio to terminate instances with the "dremio_managed" tag.
        {
          "Effect": "Allow",
          "Action": "ec2:TerminateInstances",
          "Resource": "arn:aws:ec2:*:*:instance/*",
          "Condition": {
            "StringEquals": {
              "ec2:ResourceTag/dremio_managed": "true"
            }
          }
        },
        # Require the "dremio_managed" tag for instances/volumes when creating instances.
        {
          "Effect": "Allow",
          "Action": "ec2:RunInstances",
          "Resource": [
            "arn:aws:ec2:*:*:volume/*",
            "arn:aws:ec2:*:*:instance/*"
          ],
          "Condition": {
            "StringEquals": {
              "aws:RequestTag/dremio_managed": "true"
            }
          }
        },
        # Allow creating instances without the "dremio_managed" tag on resources other than instances/volumes.
        {
          "Effect": "Allow",
          "Action": "ec2:RunInstances",
          "Resource": [
            "arn:aws:ec2:*:*:launch-template/*",
            "arn:aws:ec2:*:*:fleet/*",
            "arn:aws:ec2:*::image/*",
            "arn:aws:ec2:*:*:network-interface/*",
            "arn:aws:ec2:*:*:security-group/*",
            "arn:aws:ec2:*:*:subnet/*",
            "arn:aws:ec2:*:*:placement-group/*"
          ]
        },
        # Allow Dremio to create tags on instances/volumes only upon the initial creation of an instance.
        {
          "Effect": "Allow",
          "Action": "ec2:CreateTags",
          "Resource": [
            "arn:aws:ec2:*:*:instance/*",
            "arn:aws:ec2:*:*:volume/*"
          ],
          "Condition": {
            "StringEquals": {
              "ec2:CreateAction": "RunInstances"
            }
          }
        },
        # Allow Dremio to create tags on placement groups (PG) upon the initial creation of a PG.
        {
          "Effect": "Allow",
          "Action": "ec2:CreateTags",
          "Resource": "arn:aws:ec2:*:*:placement-group/*",
          "Condition": {
            "StringEquals": {
              "ec2:CreateAction": "CreatePlacementGroup"
            }
          }
        },
        # Allow Dremio to create tags on a launch template (LT) upon the initial creation of a LT.
        {
          "Effect": "Allow",
          "Action": "ec2:CreateTags",
          "Resource": "arn:aws:ec2:*:*:launch-template/*",
          "Condition": {
            "StringEquals": {
              "ec2:CreateAction": "CreateLaunchTemplate"
            }
          }
        },
        # Allow Dremio to create tags on a fleet upon the initial creation of the fleet.
        {
          "Effect": "Allow",
          "Action": "ec2:CreateTags",
          "Resource": "arn:aws:ec2:*:*:fleet/*",
          "Condition": {
            "StringEquals": {
              "ec2:CreateAction": "CreateFleet"
            }
          }
        },
        # Allow Dremio to create fleet only when including the "dremio_managed" tag.
        {
          "Effect": "Allow",
          "Action": "ec2:CreateFleet",
          "Resource": "arn:aws:ec2:*:*:fleet/*",
          "Condition": {
            "StringEquals": {
              "aws:RequestTag/dremio_managed": "true"
            }
          }
        },
        # Allow Dremio to create fleet with other resources without the "dremio_managed" tag.
        {
          "Effect": "Allow",
          "Action": "ec2:CreateFleet",
          "Resource": [
            "arn:aws:ec2:*:*:instance/*",
            "arn:aws:ec2:*:*:image/*",
            "arn:aws:ec2:*:*:launch-template/*",
            "arn:aws:ec2:*:*:network-interface/*",
            "arn:aws:ec2:*:*:placement-group/*",
            "arn:aws:ec2:*:*:security-group/*",
            "arn:aws:ec2:*:*:subnet/*"
          ]
        },
        # Only allow Dremio to delete fleets with the "dremio_managed" tag.
        {
          "Effect": "Allow",
          "Action": "ec2:DeleteFleets",
          "Resource": "arn:aws:ec2:*:*:fleet/*",
          "Condition": {
            "StringEquals": {
              "ec2:ResourceTag/dremio_managed": "true"
            }
          }
        },
        # Allow Dremio to create a launch template.
        {
          "Effect": "Allow",
          "Action": "ec2:CreateLaunchTemplate",
          "Resource": "arn:aws:ec2:*:*:launch-template/*"
        },
        # Only allow Dremio to delete launch templates with the "dremio_managed" tag.
        {
          "Effect": "Allow",
          "Action": "ec2:DeleteLaunchTemplate",
          "Resource": "arn:aws:ec2:*:*:launch-template/*",
          "Condition": {
            "StringEquals": {
              "ec2:ResourceTag/dremio_managed": "true"
            }
          }
        },
        # Allow Dremio to describe fleets with the "dremio_managed" tag.
        {
          "Effect": "Allow",
          "Action": "ec2:DescribeFleets",
          "Resource": "arn:aws:ec2:*:*:fleet/*",
          "Condition": {
            "StringEquals": {
              "ec2:ResourceTag/dremio_managed": "true"
            }
          }
        },
        # Only allow Dremio to delete placement groups with the "dremio_managed" tag.
        {
          "Effect": "Allow",
          "Action": "ec2:DeletePlacementGroup",
          "Resource": "arn:aws:ec2:*:*:placement-group/*",
          "Condition": {
            "StringEquals": {
              "ec2:ResourceTag/dremio_managed": "true"
            }
          }
        },
        # Allow Dremio to create a placement group.
        {
          "Effect": "Allow",
          "Action": "ec2:CreatePlacementGroup",
          "Resource": "arn:aws:ec2:*:*:placement-group/*"
        },
        # Allow Dremio to enumerate resources in the account.
        {
          "Effect": "Allow",
          "Action": [
            "ec2:DescribeImages",
            "ec2:DescribeLaunchTemplateVersions",
            "ec2:DescribeLaunchTemplates",
            "ec2:DescribeVpcs",
            "ec2:DescribeSubnets",
            "ec2:DescribeTags",
            "ec2:DescribeAvailabilityZones",
            "ec2:DescribeInstances",
            "ec2:DescribeInstanceStatus",
            "ec2:DescribeNetworkInterfaces",
            "ec2:DescribeNetworkInterfaceAttribute",
            "ec2:DescribePlacementGroups",
            "ec2:DescribeSecurityGroups",
            "ec2:DescribeVpcEndpoints",
            "ec2:DescribeVolumes"
          ],
          "Resource": "*"
        },
        # This section appears only if you chose to create a cross-account IAM role in the previous step.
        {
          "Effect": "Allow",
          "Action": [
            "iam:PassRole",
            "sts:AssumeRole"
          ],
          "Resource": [
            "<Role ARN from Step 1: Configure Storage Settings>"
          ]
        }
      ]
    }
    
    1. Copy the content of the Policy JSON text box and paste it in a location that you can access later in this procedure.
    2. To grant Dremio Cloud permissions to create and manage compute instances for Dremio engines, follow either of these two sets of steps:
    • To grant the permissions by means of an IAM user:

      a. In the Deploy to AWS using field, select Access Key.

      b. Create an IAM user with an access key.

      c. In the AWS Access Key ID field, paste the Access Key ID from the .csv file you downloaded when you created the user.

      d. In the AWS Secret Access Key field, paste the AWS Secret Access Key from the .csv file you downloaded when you created the user.

    • To grant the permissions by means of a cross-account IAM role:

      a. In the Deploy to AWS using field, select IAM Role.

      b. Create a cross-account IAM role.

      c. In the Cross-Account Role ARN field, paste the Role ARN that you copied when you created the role.

    Step 3: Configure Network Settings

    In the Network Settings section of the Cloud Connect page, specify a security group for your VPC to use with Dremio Cloud, and add an outbound rule that allows compute engines to connect to Dremio Cloud’s control plane by using TLS.

    1. Create a security group

      For steps on how to create a security group for your VPC, see create an AWS security group.

    2. In the list of security groups, select the one you created.

    3. Verify that the inbound and outbound rules are correct.

    4. Copy the security group ID and paste it in a location that you can retrieve it from in a later step.

    5. Navigate to the VPC Dashboard in the AWS Console.

    6. Locate the subnets you want to use for your Dremio Cloud account and copy their IDs. Paste them in a location that you can retrieve them from in a later step.

    7. If your organization is using AWS PrivateLink, copy the VPC endpoint ID and paste it in a location that you can retrieve it from in a later step. If your organization does not already have a VPC endpoint, follow these steps to create one.

    8. Return to the Cloud Connect page in the sign-up process for Dremio Cloud.

    9. In the Subnet field, paste the IDs of the subnets that you want to use.

    10. In the Security Group field, paste the ID of the security group that you created.

    11. If your organization is using AWS PrivateLink, paste the VPC endpoint ID into the Dremio VPC Endpoint ID field.