Creating Custom Dremio Projects

Dremio 11.0.0+ allows Dremio administrators to include their own S3 bucket, EBS volume, and EFS volume in a custom Dremio project, rather than using Dremio-provided default resources. Dremio 11.0.0+ provides an AWS configuration gateway with a REST API to create a project ID and a consistent set of AWS resource names that Dremio uses to group the resources, as well as a REST API to create the custom project with those resource inputs.

To create a Dremio custom project in AWS Edition:

  1. Generate a project ID and a name for an S3 bucket using the /aws/gateway/projectInput endpoint on your Dremio AWS Edition deployment.
  2. Create and tag the AWS resources that you want to include in your custom Dremio project.
  3. Create the custom project with the /aws/gateway/customProject endpoint on your Dremio AWS Edition deployment.

Notes

  • A single Dremio 11.0.0+ AWS Edition deployment may create only one custom Dremio project.
  • A Dremio custom project is available only while the AWS configuration gateway in Dremio 11.0.0 is running.
  • Dremio 11.0.0+ stops the AWS configuration gateway after you create a custom project or open an existing project.

Generating Project ID and an S3 Bucket Name

Dremio uses AWS tags to group your resources in a custom project. Use the projectInput endpoint to generate a project ID and a name for your S3 bucket before you create resources in AWS for your custom project.

Syntax

The endpoint requires one query parameter, the EC2 instance ID for your Dremio AWS Edition deployment.

/aws/gateway/projectInput?instanceId={EC2-instance-id}

Example Request

http://54.202.16.250:9047/aws/gateway/projectInput?instanceId=i-0941ec036ca73c48b

Example Response

{
  "statusCode": 200,
  "error": null,
  "id": "45a3cbf7-936a-4e4a-861d-17b5875a4fb2",
  "bucketName": "dremio-me-45a3cbf7-936a-4e4a-861d-17b5875a4fb2-08e9a0a2f9a8400f"
}

The id parameter in the response provides the ID for your custom Dremio project. The name of the S3 bucket in your custom project must match the value of the bucketName field.

In the next step, you will add this ID to the resources that you want to include in your custom project.

Creating and Tagging Project Resources

Dremio AWS Edition requires the following resource tags by resource type.

S3 Buckets

Tag Name Tag Value
dremio_managed true
dremio_project_id {project-id-from- projectInput-endpoint}

EBS Volumes

Tag Name Tag Value
dremio_managed true
dremio_project_id {project-id-from- projectInput-endpoint}
dremio_project_name {user-defined-string}
dremio_project_pending true

EFS Volumes

Tag Name Tag Value
dremio_managed true
dremio_project_id {project-id-from- projectInput-endpoint}

Creating a Custom Project

Create the custom Dremio project using the customProject endpoint of the AWS configuration gateway.

Syntax

The customProject endpoint requires the following parameters in the HTTP request body:

Parameter Data Type Description
name String AWS Project Name
id String Custom AWS Project ID
engineSize String One of the following values: SMALL, MEDIUM, LARGE, XX_LARGE, XXX_LARGE, and CUSTOM
enableAutoBackups Boolean true or false
engineNodeType String One of the following values: STANDARD, HIGH_MEMORY, HIGH_CPU, HIGH_CAPACITY, and CUSTOM
iamInstanceProfile String Must be an empty string: ""
instanceId String The EC2 instance ID that you passed to the aws/gateway/projectInput endpoint
disablePublicIp Boolean true or false
ebsVolumeId String AWS identifier for an EBS volume to include in the custom project
efsVolumeName String AWS identifier for an EFS volume to include in the custom project
s3BucketName String AWS identifier for an S3 bucket to include in the custom project

Example Request

The following example creates a custom project with one EBS volume, one EFS volume, and one S3 bucket:

{
	"name": "MyProject",
	"id": "45a3cbf7-936a-4e4a-861d-17b5875a4fb2",
	"engineSize": "SMALL",
	"customNodeCount": 0,
	"enableAutoBackups": false,
	"engineNodeType": "STANDARD",
	"iamInstanceProfile": "",
	"instanceId": "i-0941ec036ca73c48b",
	"disablePublicIp": true,
	"ebsVolumeId": "vol-0f8064167d3d6cb98",
	"efsVolumeName": "fs-5a522a5f",
	"s3BucketName": "dremio-me-45a3cbf7-936a-4e4a-861d-17b5875a4fb2-08e9a0a2f9a8400f"
}

Example Response

The following is a successful response:

{
"statusCode": 200,
"error": null,
"instanceId": "i-0c749fab209c8fc68",
"id": "ba9277bb-41ea-4374-a64f-3b67c862de6d",
"name": "Custom1",
"iamInstanceProfile": "",
"engineNodeType": "STANDARD",
"engineSize": "MEDIUM",
"customNodeCount": 0,
"customEngineType": null,
"disablePublicIp": true,
"enableAutoBackups": true,
"ebsVolumeId": "vol-09c318e4b4a86975e",
"efsVolumeName": "fs-9e145c9b",
"s3BucketName": "dremio-me-ba9277bb-41ea-4374-a64f-3b67c862de6d-3c3a5327b8753a81"
}