Skip to main content

Dremio Cloud Troubleshooting for AWS

This guide can help you troubleshoot issues that you encounter at any stage from configuring your AWS account with Dremio Cloud to running AWS EC2 instances.

Project Creation Failing​

Problem: Your project creation fails during the CFT or manual setup, and you see this error: Received response status [FAILED] from custom resource. Message returned: Reason: INVALID_ARGUMENT: Unable to create project <project-name>. Cannot access the project store. Check the path (<path>) and credentials (<credentials>) are valid. Please go back to Dremio UI and retry. (<RequestId>)

Solution: This issue occurs if an STS endpoint is not enabled for the us-west-2 region, which is required for the Dremio control plane to connect to your cloud account. Verify that the STS endpoint for us-west-2 in your IAM dashboard is enabled.

For regions that must be enabled, AWS activates STS automatically when you enable the region. After you enable a region, AWS STS is always active for the region and you cannot deactivate it. To learn how to enable a region, see Managing AWS Regions.

To activate AWS STS in a region that is enabled by default through the IAM console, complete the following steps:

  1. Open the IAM console and log in as a root user or an IAM user with permissions to perform IAM administration tasks.

  2. Choose Account settings in the side navigation pane.

  3. Under Security Token Service (STS), find the region that you want to activate the STS endpoint.

  4. Toggle the STS status to activate.

  5. Go back to the CFT or manual process in the Dremio UI to retry the project creation.

For more information, see Managing AWS STS in an AWS Region.

EC2 Instances Are Not Launching​

Problem: The Dremio Console shows an engine is starting but the EC2 instances are not starting in the AWS Management Console and you receive an error message that the instance launch failed.

Solution: Check in the AWS Management Console if you are missing the following settings:

  • DNS hostnames and resolutions are enabled on your VPC
  • Auto-assign public IPv4 address are enabled on your public subnets
  • Route table routed to an internet gateway for public subnets or a NAT gateway for private subnets

Enabling DNS Hostnames and Resolution​

To enable the DNS hostnames and resolution:

  1. Open the Your VPCs page in a new tab.
  2. Select your VPC by checking the box next to it.
  3. Go to Actions > Edit VPC Settings.
  4. For Enable DNS hostnames, check the box to enable the setting.
  5. For Enable DNS resolution, check the box to enable the setting.
  6. Click Save.
  7. Go back to the Dremio Console to retry starting the engine.

Enabling the Auto-Assign Public IPv4 Address​

To enable the auto-assign public IPv4 address:

  1. Open the VPC Dashboard in a new tab and click Subnets in the side navigation bar.
  2. Select your subnet by checking the box next to it.
  3. Go to Actions > Edit subnet settings.
  4. Check the box next to Enable auto-assign public IPv4 address to enable.
  5. Click Save.
  6. Go back to the Dremio Console to retry starting the engine.

Routing to an Internet Gateway​

To route table to an internet gateway:

  1. Open the VPC Dashboard in a new tab and click Route tables in the side navigation bar.

  2. On the Route tables page, select your route table by checking the box next to it.

  3. Go to Actions > Edit routes

  4. On the Edit routes page, click the Add route button and set the following:

    a. For Destination, select 0.0.0.0/0.

    b. For Target, select Internet Gateway. Then select your internet gateway from the dropdown.

  5. Click Save changes.

  6. Go back to the Dremio Console to retry starting the engine.

Routing to a NAT Gateway​

To route table to a NAT gateway:

  1. Open the VPC Dashboard in a new tab and click Route tables in the side navigation bar.

  2. On the Route tables page, select your route table by checking the box next to it.

  3. Go to Actions > Edit routes

  4. On the Edit routes page, click the Add route button and set the following:

    a. For Destination, select 0.0.0.0/0.

    b. For Target, select NAT Gateway. Then select your NAT gateway from the dropdown.

note

You will need to use a different destination CIDR block.

  1. Click Save changes.
  2. Go back to the Dremio Console to retry starting the engine.

For steps on how to create or configure an internet gateway, NAT gateway, route table, or other resources, see AWS Resources.

Running EC2 Instances Fails or Engine Scaling Fails​

Problem: When running an instance fails or the engine scaling fails, you see this error: You have requested more vCPU capacity than your current vCPU limit of <limit> allows for the instance bucket that the specified instance type belongs to. Please visit http://aws.amazon.com/contact-us/ec2-request to request an adjustment to this limit.

Solution: The vCPU limit is a limit imposed by AWS and varies by account, so you will need to log in to your AWS account and submit a request for a service limit increase.

To submit a service limit increase:

  1. Determine the number of vCPUs that you need. See example calculations.
  2. Go to http://aws.amazon.com/contact-us/ec2-request and log in to your AWS account.
  3. For Limit type, choose EC2 Instances.
  4. For Region, select your region. Because EC2 service quotas affect one region at a time, you must request service quota increases in each AWS Region where you need more resources.
  5. Choose your primary instance type and set the new limit value.
  6. For Case description, enter a description for your use case.
  7. Click Submit.