Configuring Cloud Resources Manually

If you are creating your first Dremio Sonar project and would like to configure cloud resources manually, then you will need to set up storage, compute, and network access.

note

Before you begin, review the Prerequisites for Configuring Cloud Resources.

AWS

note

If you have already completed the Set Up Project page in the Add Sonar Project dialog, proceed to Configure Resources: Set Up Storage Access.

Set Up a Project

In the Set Up Project section, complete the following steps:

1. On your organization homepage, click Add Sonar Project on the Sonar card.

2. In the Add Sonar Project dialog, for Project name, specify a name to identify your project. You can change the name of the project after creation.

3. For Project Name, enter a name to identify this project. This name can be changed after project creation.

4. For AWS Region, select the region in which Dremio will create and manage engines used to execute queries. For a list of supported regions, reference Supported Regions.

5. (Optional) For AWS Tags, add the AWS tags to identify the compute resources created for this project.

6. Click Next.

7. On the Configure Resources page, click Create project manually to create the AWS resources and grant Dremio permissions manually. Follow the steps in the next section to configure cloud resources.

Set Up Storage Access

note

If you encounter an issue during the configuration, see Dremio Cloud Troubleshooting for AWS.

In the Set Up Storage Access section of the Configure Resources page, configure the S3 bucket that will be used as the metadata store for your project.

When you configure storage settings, you grant Dremio Cloud read and write permissions to the S3 bucket.

To configure storage settings:

1. Create an Amazon S3 bucket. For steps on how to create an S3 bucket in the AWS Console, see AWS: Creating a Dremio Cloud Project Store.

2. For Project Store, paste the path of the S3 bucket. This fills in the PROJECT-STORE-BUCKET-NAME parameter in the policy JSON. The rest of the policy JSON sets the minimum policy requirement to allow Dremio Cloud read and write access to the project store.

3. To grant Dremio read and write permissions to the project store, follow either of these two sets of steps:

   • To grant the permissions by means of an IAM user:

     a. For Authentication Method, select Access Key.

     b. Copy the policy JSON from the dialog and paste it into a location that you can access later. To understand what permissions the policy JSON grants, see Policy JSON: Setting Up Storage Access.

     c. In the AWS Console, create an IAM user with an access key.

     d. For Access Key, paste the Access Key ID from the .csv file you downloaded when you created the user.

     e. For Secret Access Key, paste the AWS Secret Access Key from the .csv file you downloaded when you created the user.

   • To grant the permissions by means of a cross-account IAM role:

     a. For Authentication Method, select IAM Role.

     b. Copy the policy JSON from the dialog and paste it into a location that you can access later.

     c. Copy this JSON snippet and paste it into a location that you can access later.

     Trust Policy for IAM Role

       {
         "Version": "2012-10-17",
         "Statement": [
           {
             "Effect": "Allow",
             "Principal": {
               "Service": "ec2.amazonaws.com"
             },
             "Action": "sts:AssumeRole"
           },
           {
             "Effect": "Allow",
             "Principal": {
               "AWS": "arn:aws:iam::<trust account ID>:root"
             },
             "Action": "sts:AssumeRole",
             "Condition": {
               "StringEquals": {
                 "sts:ExternalId": "<external ID>"
               }
             }
           }
         ]
       }

     d. Copy the Trust Account ID and the External ID and paste them into the JSON that you copied in the previous step.

     d. In the AWS Console, create a cross-account IAM role.

     e. For Role ARN, paste the cross-account role ARN that you copied when you created the role.

     f. For Instance Profile ARN, paste the instance profile ARN that you copied when you created the role.

     g. Click Validate.

Set Up Compute Access

In the Set Up Compute Access section of the Configure Resources page, specify the credentials for that will be used to create and manage compute engines in your Amazon VPC.

When you configure compute settings, you grant Dremio Cloud permissions to create and manage compute instances for Dremio engines. As with the storage settings, you can grant the permissions by means of an IAM user or an IAM role.

note

You can use the same IAM user that you created for granting access to storage settings by adding an additional JSON policy to the existing IAM user. For steps on how to attach new permission policies to an existing IAM user, see setting up AWS permissions.

Permissions are granted only on resources that are tagged as dremio_managed.

1. Copy the content of the Policy JSON text box and paste it in a location that you can access later. To understand the permissions the policy JSON grants, see Policy JSON: Set up Compute Access.
2. To grant Dremio Cloud permissions to create and manage compute instances for Dremio engines, follow either of these two sets of steps:

• To grant the permissions by means of an IAM user:

  a. For AWS Deploy Method, select Access Key.

  b. Create an IAM user with an access key.

  c. For Access Key, paste the Access Key ID from the .csv file you downloaded when you created the user.

  d. For Secret Access, paste the AWS Secret Access Key from the .csv file you downloaded when you created the user.

• To grant the permissions by means of a cross-account IAM role:

  a. For AWS Deploy Method, select IAM Role.

  b. Create a cross-account IAM role.

  c. For Role ARN, paste the Role ARN that you copied when you created the role.

Set Up Network Access

In the Set Up Network Access section of the Configure Resources page, specify a security group for your VPC to use with Dremio Cloud, and add an outbound rule that allows compute engines to connect to Dremio Cloud's control plane by using TLS.

1. Create a security group.

   For steps on how to create a security group for your VPC, see create an AWS security group.

2. In the list of security groups, select the one you created.

3. Verify that the inbound and outbound rules are correct.

4. Copy the security group ID and paste it in a location that you can retrieve it from in a later step.

5. Navigate to the VPC Dashboard in the AWS Console.

6. Locate the subnets you want to use for your Dremio Cloud account and copy their IDs. Paste them in a location that you can retrieve them from in a later step. Guidelines for choosing subnets are located here.

7. To use AWS PrivateLink, copy the VPC endpoint ID and paste it in a location that you can retrieve it from in a later step. If your organization does not already have a VPC endpoint, follow these steps to create one.

8. Return to the Configure resources page in the Add Sonar Project dialog for Dremio Cloud.

9. For Subnets, paste the IDs of the subnets that you want to use.

10. For Security Group, paste the ID of the security group that you created.

11. (Optional) For Dremio VPC Endpoint ID, paste the VPC endpoint ID if you are using AWS PrivateLink.

Azure

1. On your organization homepage, click Add Sonar Project on the Sonar card.

2. In the Add Sonar Project dialog, for Project name, specify a name to identify your project. You can change the name of the project after creation.

3. For Arctic catalog name, specify a name to identify the primary catalog in your project. The name cannot be changed once the catalog has been created. The name cannot include the following special characters: /, :, [, or ].

4. Select a cloud provider.

5. Select the region in which compute resources and the project store will be created. For a list of supported regions, see Supported Regions.

6. (Optional) Add one or more tags to help you identify compute resources for this project in your cloud provider account.

7. Click Next.

8. Click Create project manually.

9. Under 1 - Set Up Storage Access:

   1. For Tenant ID, paste the ID of the Azure tenant.

   2. For Project Store, paste the name of the storage container that you want to use for the project store.

   3. For Account Name, paste the name of the storage account that you want to use for the project store.

   4. For Client ID, paste the application (client) ID of the registered application.

   5. For Client Secret, paste the value of the client secret that you created in the registered application.

   6. Click Validate to confirm that the project store credentials are valid.

10. Under 2 - Set Up Compute Access:

    1. For Tenant ID, paste the ID of the Azure tenant.

    2. For Subscription ID, paste the ID of your Azure subscription.

    3. For Client ID, paste the application (client) ID of the registered application.

    4. For Client Secret, paste the value of the client secret that you created in the registered application.

    5. For Resource Group, paste the name of the resource group.

11. Under 3 - Set Up Network Access:

    1. For Subnet, paste the name of the subnet.

    2. (Optional) For Network Security Group (Optional), paste the name of the security network group.

    3. For Virtual Network, paste the name of the virtual network.

    4. (Optional) For Dremio Private Endpoint Name (Optional), enter the name of your private endpoint. For more information, see Azure Private Link.

    5. Click Validate.

12. Click Done.

It will take 4-5 minutes for your project to complete the setup process.