Skip to main content

Dremio Cloud Troubleshooting for AWS

This guide can help you troubleshoot issues that you encounter at any stage from configuring your AWS account with Dremio Cloud to running AWS EC2 instances.

Project Creation Failing​

Problem
Your project creation fails during the CFT or manual setup, and you see this error:

Received response status [FAILED] from custom resource. Message returned: Reason: INVALID_ARGUMENT: Unable to create project <project-name>. Cannot access the project store. Check the path (<path>) and credentials (<credentials>) are valid. Please go back to Dremio UI and retry. (<RequestId>)

Solution
This issue occurs if an STS endpoint is not enabled for the us-west-2 region, which is required for the Dremio control plane to connect to your cloud account. Verify that the STS endpoint for us-west-2 in your IAM dashboard is enabled.

For regions that must be enabled, AWS activates STS automatically when you enable the region. After you enable a region, AWS STS is always active for the region and you cannot deactivate it. To learn how to enable a region, see Managing AWS Regions.

To activate AWS STS in a region that is enabled by default through the IAM console, complete the following steps:

  1. Open the IAM console and log in as a root user or an IAM user with permissions to perform IAM administration tasks.

  2. Choose Account settings in the side navigation pane.

  3. Under Security Token Service (STS), find the region that you want to activate the STS endpoint.

  4. Toggle the STS status to activate.

  5. Go back to the CFT or manual process in the Dremio UI to retry the project creation.

For more information, see Managing AWS STS in an AWS Region.

EC2 Instances Not Launching​

Problem
The Dremio console shows an engine is starting but the EC2 instances are not starting in the AWS Management Console and you receive an error message that the instance launch failed.

Solution
In the AWS Management Console, check the cases below and follow the respective steps to fix them:

  • DNS hostnames and resolutions are not enabled on your VPC

    To enable the DNS hostnames and resolutions do the following:

    1. Open the Your VPCs page in a new tab.
    2. Select your VPC by checking the box next to it.
    3. Go to Actions > Edit VPC Settings.
    4. For Enable DNS hostnames, check the box to enable the setting.
    5. For Enable DNS resolution, check the box to enable the setting.
    6. Click Save.
    7. Go back to the Dremio Console to retry starting the engine.

  • Auto-assign of public IPv4 addresses is not enabled on your public subnets

    To enable the auto-assign public IPv4 address:

    1. Open the VPC Dashboard in a new tab and click Subnets in the side navigation bar.
    2. Select your subnet by checking the box next to it.
    3. Go to Actions > Edit subnet settings.
    4. Check the box next to Enable auto-assign public IPv4 address to enable.
    5. Click Save.
    6. Go back to the Dremio Console to retry starting the engine.

  • No route table to an internet gateway (for public subnets)

    To route the table to an internet gateway:

    1. Open the VPC Dashboard in a new tab and click Route tables in the side navigation bar.

    2. On the Route tables page, select your route table by checking the box next to it.

    3. Go to Actions > Edit routes

    4. On the Edit routes page, click the Add route button and set the following:

      a. For Destination, select 0.0.0.0/0.

      b. For Target, select Internet Gateway. Then select your internet gateway from the dropdown.

    5. Click Save changes.

    6. Go back to the Dremio console to retry starting the engine.

  • No route table to an NAT gateway (for private subnets)

    To route the table to a NAT gateway:

    1. Open the VPC Dashboard in a new tab and click Route tables in the side navigation bar.

    2. On the Route tables page, select your route table by checking the box next to it.

    3. Go to Actions > Edit routes

    4. On the Edit routes page, click the Add route button and set the following:

      a. For Destination, select 0.0.0.0/0.

      b. For Target, select NAT Gateway. Then select your NAT gateway from the dropdown.

      note

      You will need to use a different destination CIDR block.

    5. Click Save changes.

    6. Go back to the Dremio Console to retry starting the engine.

For steps on how to create or configure an internet gateway, NAT gateway, route table, or other resources, see AWS Resources.

Running EC2 Instances or Engine Scaling Failing​

Problem
When running an instance fails or the engine scaling fails, you see this error:

You have requested more vCPU capacity than your current vCPU limit of <limit> allows for the instance bucket that the specified instance type belongs to. Please visit http://aws.amazon.com/contact-us/ec2-request to request an adjustment to this limit.

Solution
The vCPU limit is a limit imposed by AWS and varies by account, so you will need to log in to your AWS account and submit a request for a service limit increase.

To submit a service limit increase:

  1. Determine the number of vCPUs that you need. See example calculations.
  2. Go to http://aws.amazon.com/contact-us/ec2-request and log in to your AWS account.
  3. For Limit type, choose EC2 Instances.
  4. For Region, select your region. Because EC2 service quotas affect one region at a time, you must request service quota increases in each AWS Region where you need more resources.
  5. Choose your primary instance type and set the new limit value.
  6. For Case description, enter a description for your use case.
  7. Click Submit.