Skip to main content

Managing Clouds

A cloud represents a virtual environment in which Dremio Cloud engines run and where the project metadata store is created and maintained. A single cloud can be associated with many projects.

A cloud object is associated with a single region in your tenant. For information on supported regions and availability zones, see the Supported Regions page.

Viewing All Clouds​

To view the clouds that are in a Dremio Cloud organization:

  1. Click This is the icon that represents the Organization. in the side navigation bar.
  2. Click This is the icon that represents the Organization settings. in the side navigation bar.
  3. Select Clouds in the organization settings sidebar.

Adding a Cloud​

note

This procedure uses the CloudFormation template (CFT), which is the recommended method for configuring cloud resources. For more information on the resources being created and the permissions granted to Dremio, see the annotated CFT.

Follow the steps below to add a new cloud to your organization:

  1. In the top-right corner of the Clouds page, click Add Cloud.

  2. Select a cloud provider.

To continue the configuration, follow the steps for your cloud provider.

  1. In the Add AWS Cloud dialog, for Cloud Name, specify a name for your cloud that is unique within your organization.

  2. For Region, enter the region associated with the cloud account you are connecting to Dremio Cloud. For more information, see supported regions.

  3. Click Launch CloudFormation Template, which will open the AWS Console in a new browser tab for next steps. If not already authenticated, sign in to your AWS account on this page.

  4. On the "Quick create stack" page in the AWS Console, for Stack name, enter a name to identify the stack. This name must be unique to your account and cannot include underscores.

  5. For VPC, select a VPC in which the compute resources (EC2 instances) will be created.

  6. For Subnets, select one or more subnets within the VPC. You can select multiple subnets across different availability zones.

  7. Select I acknowledge that AWS CloudFormation might create IAM resources.

  8. Click Create stack.

Stack creation can take up to four minutes. When stack creation is complete, the required cloud resources have been configured successfully and you can return to the browser tab running Dremio Cloud to continue.

Adding an AWS Cloud Manually​

Instead of using the CFT, you can configure cloud resources manually for AWS.

To add a new cloud manually:

  1. In the top-right corner of the Clouds page, click Add Cloud.

  2. Select AWS. A dialog opens for adding an AWS cloud.

  3. In the Add AWS Cloud dialog, for Cloud Name, specify a name for your cloud that is unique within your organization.

  4. For Region, enter the region associated with the cloud account you are connecting to Dremio Cloud. For more information, see supported regions.

  5. Click the link at the bottom of the dialog to create the AWS resources and grant Dremio permissions manually.

  6. In the Compute Access section, you give Dremio Cloud access to your cloud. Follow these steps:

    a. Copy the policy JSON to your clipboard. This JSON lists the tasks that Dremio Cloud needs permission to carry out on your cloud. You use the policy JSON in the AWS Console in step 7.

    b. For Compute Credentials, select the security method that you want to use to give Dremio Cloud the permission it needs.

  7. Log in to the AWS Console, and then follow either of these steps:

    a. If you selected Access Key in step 6, then create an AWS IAM user and download the access key ID and secret access key.

    b. If you selected IAM Role in step 6, then create an AWS IAM role. As you create the role, return to Dremio Cloud to copy the trust account ID and the external ID to your clipboard when you need them.

  8. Return to the Add AWS Cloud dialog and follow either of these steps:

    a. If you selected Access Key in step 6, paste into the appropriate fields the AWS access key ID and AWS secret access key that you downloaded after creating an IAM User in step 7.

    b. If you selected IAM Role in step 6, enter the Cross-Account Role ARN of the IAM Role you created in step 7.

  9. In the AWS Console, create a security group that allows engines to connect to Dremio Cloud over TLS.

  10. In the Network Access section, follow these steps:

    a. For Subnets, specify the subnet ID associated with your Amazon Virtual Private Cloud (Amazon VPC). If you want to use more than one subnet, click Add Subnet once for each additional subnet, and then specify the ID of one subnet per added field. When Dremio Cloud needs to start up query engines, it searches the subnets for resources it can use to spin up required instances. Adding more subnets increases the chances of finding resources, should one or more of the availability zones associated with a subnet run out of instances required by Dremio Cloud.

    note

    Guidelines for adding subnets:

    • Ensure that subnet IDs are unique across all of the availability zones within an Amazon VPC.

    • Ensure that each subnet that you specify belongs to a separate availability zone; for example, if you specify subnet A and subnet B, they cannot both be in availability zone C, but must be in separate availability zones.

    • Specify only private subnets or only public subnets; mixing private and public subnets is not supported.

    b. For Security Group, specify the ID of the security group that you created in step 9.

  11. (Optional) Enter the VPC endpoint ID if you are using AWS PrivateLink.

  12. Click Add.

Editing a Cloud​

When you edit a cloud, you can change the following settings depending on your cloud provider:

caution

Modifying the subnet ID restarts all affected engines and replicas with the new configuration after the drain time expires.

To edit a cloud:

  1. On the Clouds page, hover over the row of the cloud and click This is the icon that represents the Edit settings. that appears next to the cloud.

  2. In the Edit Cloud dialog, make changes to the cloud settings.

  3. Click Save.

Deleting a Cloud​

To delete a cloud from a Dremio Cloud organization:

caution

To delete a cloud you must first delete or select a new cloud for the engines that are associated with this cloud.

  1. On the Clouds page, hover over the row of the cloud and click the Delete This is the icon that represents the Delete settings. icon that appears next to the cloud.
  2. Confirm that you want to delete the cloud.