Managing Clouds

A cloud represents a virtual environment in which Dremio Cloud engines run and where the project metadata store is created and maintained. A single cloud can be associated with many projects.

A cloud object is associated with a single region in your tenant. For information on supported regions and availability zones, see the Supported Regions page.

Viewing All Clouds

To view the clouds that are in a Dremio Cloud organization:

1. Click in the side navigation bar.
2. Click in the side navigation bar.
3. Select Clouds in the organization settings sidebar.

Adding a Cloud

note

This procedure uses the CloudFormation template (CFT), which is the recommended method for configuring cloud resources. For more information on the resources being created and the permissions granted to Dremio, see the annotated CFT.

Follow the steps below to add a new cloud to your organization:

1. In the top-right corner of the Clouds page, click Add Cloud.

2. Select a cloud provider.

To continue the configuration, follow the steps for your cloud provider.

AWS

In the Add AWS Cloud dialog:

3. For Cloud Name, specify a name for your cloud that is unique within your organization.


4. For Region, enter the region associated with the cloud account you are connecting to Dremio Cloud. For more information, see supported regions.


5. Click Launch CloudFormation Template, which will open the AWS Console in a new browser tab for next steps. If not already authenticated, sign in to your AWS account on this page.


6. On the "Quick create stack" page in the AWS Console, for Stack name, enter a name to identify the stack. This name must be unique to your account and cannot include underscores.


7. For VPC, select a VPC in which the compute resources (EC2 instances) will be created.


8. For Subnets, select one or more subnets within the VPC. You can select multiple subnets across different availability zones.


9. Select I acknowledge that AWS CloudFormation might create IAM resources.


10. Click Create stack.

Stack creation can take up to four minutes. When stack creation is complete, the required cloud resources have been configured successfully and you can return to the browser tab running Dremio Cloud to continue.

Azure

In the Add Azure Cloud dialog:

3. For Cloud Name, specify a name for your cloud that is unique within your organization.


4. For Region, enter the region associated with the cloud account you are connecting to Dremio Cloud. For more information, see supported regions.


5. For Tenant ID, enter the ID of the Azure tenant.


6. For Subscription ID, enter your subscription ID.


7. For Application (client) ID, enter the application (client) ID of the registered application.


8. For Client secret, enter the value of the client secret that you created in the registered application.


9. For Compute resource group, enter the name of the Compute resource group. Learn more about creating or using a resource group in Azure.


10. (Optional) If you want to assign custom privileges for roles, copy the Compute policy JSON to your clipboard. This JSON lists the tasks that Dremio Cloud needs permission to carry out on your cloud.


11. Under Network Access, the checkbox Use the same resource group as in Compute Access is ticked by default. This means that the Compute resource group is the default resource group where VMSS will be scaled, and all network resources will be present.
    If you untick the checkbox, you can specify a separate Network resource group (distinct from the Compute resource group) focused on networking resources:
    
    
    1. A Network resource group input is available to enter the name of the Network resource group. Learn more about creating or using a resource group in Azure.
    
    
    2. (Optional) If you want to assign custom privileges for roles, copy the Network policy JSON (at the bottom of the form) to your clipboard. This JSON lists the tasks that Dremio Cloud needs permission to carry out on your cloud.


12. For Subnet, enter the name of the subnet.


13. (Optional) Enter the name of the Network security group that you want this cloud to use.


14. For Virtual Network, enter the name of the virtual network.


15. (Optional) For Dremio private endpoint name, enter the name here if you created an outbould private endpoint.


16. Click Add.

Adding an AWS Cloud Manually

Instead of using the CFT, you can configure cloud resources manually for AWS.

To add a new cloud manually:

1. In the top-right corner of the Clouds page, click Add Cloud.

2. Select AWS. A dialog opens for adding an AWS cloud.

3. In the Add AWS Cloud dialog, for Cloud Name, specify a name for your cloud that is unique within your organization.

4. For Region, enter the region associated with the cloud account you are connecting to Dremio Cloud. For more information, see supported regions.

5. Click the link at the bottom of the dialog to create the AWS resources and grant Dremio permissions manually.

6. In the Compute Access section, you give Dremio Cloud access to your cloud. Follow these steps:

a. Copy the policy JSON to your clipboard. This JSON lists the tasks that Dremio Cloud needs permission to carry out on your cloud. You use the policy JSON in the AWS Console in step 7.

b. For Compute Credentials, select the security method that you want to use to give Dremio Cloud the permission it needs.

7. Log in to the AWS Console, and then follow either of these steps:

a. If you selected Access Key in step 6, then create an AWS IAM user and download the access key ID and secret access key.

b. If you selected IAM Role in step 6, then create an AWS IAM role. As you create the role, return to Dremio Cloud to copy the trust account ID and the external ID to your clipboard when you need them.

8. Return to the Add AWS Cloud dialog and follow either of these steps:

a. If you selected Access Key in step 6, paste into the appropriate fields the AWS access key ID and AWS secret access key that you downloaded after creating an IAM User in step 7.

b. If you selected IAM Role in step 6, enter the Cross-Account Role ARN of the IAM Role you created in step 7.

9. In the AWS Console, create a security group that allows engines to connect to Dremio Cloud over TLS.

10. In the Network Access section, follow these steps:

a. For Subnets, specify the subnet ID associated with your Amazon Virtual Private Cloud (Amazon VPC). If you want to use more than one subnet, click Add Subnet once for each additional subnet, and then specify the ID of one subnet per added field. When Dremio Cloud needs to start up query engines, it searches the subnets for resources it can use to spin up required instances. Adding more subnets increases the chances of finding resources, should one or more of the availability zones associated with a subnet run out of instances required by Dremio Cloud.

note

Guidelines for adding subnets:

• Ensure that subnet IDs are unique across all of the availability zones within an Amazon VPC.

• Ensure that each subnet that you specify belongs to a separate availability zone; for example, if you specify subnet A and subnet B, they cannot both be in availability zone C, but must be in separate availability zones.

• Specify only private subnets or only public subnets; mixing private and public subnets is not supported.

b. For Security Group, specify the ID of the security group that you created in step 9.

11. (Optional) Enter the VPC endpoint ID if you are using AWS PrivateLink.

12. Click Add.

Editing a Cloud

When you edit a cloud, you can change the following settings depending on your cloud provider:

AWS

• Cloud name
• AWS access key ID and AWS secret access key
• IAM role and AWS cross-account role ARN
• Subnet IDs
• Security group ID
• VPC endpoint ID (for AWS PrivateLink configurations)

Azure

• Cloud name
• Subscription ID
• Application (client) ID and client secret
• Tenant ID
• Subnet name
• Network security group
• Virtual network

caution

Modifying the subnet ID restarts all affected engines and replicas with the new configuration after the drain time expires.

To edit a cloud:

1. On the Clouds page, hover over the row of the cloud and click that appears next to the cloud.

2. In the Edit Cloud dialog, make changes to the cloud settings.

3. Click Save.

Deleting a Cloud

To delete a cloud from a Dremio Cloud organization:

caution

To delete a cloud you must first delete or select a new cloud for the engines that are associated with this cloud.

1. On the Clouds page, hover over the row of the cloud and click the Delete icon that appears next to the cloud.
2. Confirm that you want to delete the cloud.