Skip to main content
Version: current [24.3.x]

Creating Custom Dremio Projects

Dremio 11.0.0+ allows Dremio administrators to include their own S3 bucket, EBS volume, and EFS volume in a custom Dremio project, rather than using Dremio-provided default resources. Dremio 11.0.0+ provides an AWS configuration gateway with a REST API to create a project ID and a consistent set of AWS resource names that Dremio uses to group the resources, as well as a REST API to create the custom project with those resource inputs.

To create a Dremio custom project in AWS Edition:

  1. Generate a project ID and a name for an S3 bucket using the /aws/gateway/projectInput endpoint on your Dremio AWS Edition deployment.
  2. Create and tag the AWS resources that you want to include in your custom Dremio project.
  3. Create the custom project with the /aws/gateway/customProject endpoint on your Dremio AWS Edition deployment.
note
  • A single Dremio 11.0.0+ AWS Edition deployment may create only one custom Dremio project.
  • A Dremio custom project is available only while the AWS configuration gateway in Dremio 11.0.0 is running.
  • Dremio 11.0.0+ stops the AWS configuration gateway after you create a custom project or open an existing project.

Generating Project ID and an S3 Bucket Name

Dremio uses AWS tags to group your resources in a custom project. Use the projectInput endpoint to generate a project ID and a name for your S3 bucket before you create resources in AWS for your custom project.

Syntax

The endpoint requires one query parameter, the EC2 instance ID for your Dremio AWS Edition deployment.

Query parameter syntax
/aws/gateway/projectInput?instanceId={EC2-instance-id}

Example Request

Example requst
http://54.202.16.250:9047/aws/gateway/projectInput?instanceId=i-0941ec036ca73c48b

Example Response

Example response
{
"statusCode": 200,
"error": null,
"id": "45a3cbf7-936a-4e4a-861d-17b5875a4fb2",
"bucketName": "dremio-me-45a3cbf7-936a-4e4a-861d-17b5875a4fb2-08e9a0a2f9a8400f"
}

The id parameter in the response provides the ID for your custom Dremio project. The name of the S3 bucket in your custom project must match the value of the bucketName field.

In the next step, you will add this ID to the resources that you want to include in your custom project.

Creating and Tagging Project Resources

Dremio AWS Edition requires the following resource tags by resource type.

S3 Buckets

Tag NameTag Value
dremio_managedtrue
dremio_project_id{project-id-from- projectInput-endpoint}

EBS Volumes

Tag NameTag Value
dremio_managedtrue
dremio_project_id{project-id-from- projectInput-endpoint}
dremio_project_name{user-defined-string}
dremio_project_pendingtrue

EFS Volumes

Tag NameTag Value
dremio_managedtrue
dremio_project_id{project-id-from- projectInput-endpoint}

Creating a Custom Project

Create the custom Dremio project using the customProject endpoint of the AWS configuration gateway.

Syntax

The customProject endpoint requires the following parameters in the HTTP request body:

ParameterData TypeDescription
nameStringAWS Project Name
idStringCustom AWS Project ID
engineSizeStringOne of the following values: SMALL, MEDIUM, LARGE, XX_LARGE, XXX_LARGE, and CUSTOM
enableAutoBackupsBooleantrue or false
engineNodeTypeStringOne of the following values: STANDARD, HIGH_MEMORY, HIGH_CPU, HIGH_CAPACITY, and CUSTOM
iamInstanceProfileStringMust be an empty string: ""
instanceIdStringThe EC2 instance ID that you passed to the aws/gateway/projectInput endpoint
disablePublicIpBooleantrue or false
ebsVolumeIdStringAWS identifier for an EBS volume to include in the custom project
efsVolumeNameStringAWS identifier for an EFS volume to include in the custom project
s3BucketNameStringAWS identifier for an S3 bucket to include in the custom project

Example Request

The following example creates a custom project with one EBS volume, one EFS volume, and one S3 bucket:

Example request
{
"name": "MyProject",
"id": "45a3cbf7-936a-4e4a-861d-17b5875a4fb2",
"engineSize": "SMALL",
"customNodeCount": 0,
"enableAutoBackups": false,
"engineNodeType": "STANDARD",
"iamInstanceProfile": "",
"instanceId": "i-0941ec036ca73c48b",
"disablePublicIp": true,
"ebsVolumeId": "vol-0f8064167d3d6cb98",
"efsVolumeName": "fs-5a522a5f",
"s3BucketName": "dremio-me-45a3cbf7-936a-4e4a-861d-17b5875a4fb2-08e9a0a2f9a8400f"
}

Example Response

The following is a successful response:

Example response
{
"statusCode": 200,
"error": null,
"instanceId": "i-0c749fab209c8fc68",
"id": "ba9277bb-41ea-4374-a64f-3b67c862de6d",
"name": "Custom1",
"iamInstanceProfile": "",
"engineNodeType": "STANDARD",
"engineSize": "MEDIUM",
"customNodeCount": 0,
"customEngineType": null,
"disablePublicIp": true,
"enableAutoBackups": true,
"ebsVolumeId": "vol-09c318e4b4a86975e",
"efsVolumeName": "fs-9e145c9b",
"s3BucketName": "dremio-me-ba9277bb-41ea-4374-a64f-3b67c862de6d-3c3a5327b8753a81"
}