Creating Custom Dremio Projects

Dremio 11.0.0+ allows Dremio administrators to include their own S3 bucket, EBS volume, and EFS volume in a custom Dremio project, rather than using Dremio-provided default resources. Dremio 11.0.0+ provides an AWS configuration gateway with a REST API to create a project ID and a consistent set of AWS resource names that Dremio uses to group the resources, as well as a REST API to create the custom project with those resource inputs.

To create a Dremio custom project in AWS Edition:

1. Generate a project ID and a name for an S3 bucket using the /aws/gateway/projectInput endpoint on your Dremio AWS Edition deployment.
2. Create and tag the AWS resources that you want to include in your custom Dremio project.
3. Create the custom project with the /aws/gateway/customProject endpoint on your Dremio AWS Edition deployment.

note

• A single Dremio 11.0.0+ AWS Edition deployment may create only one custom Dremio project.
  
• A Dremio custom project is available only while the AWS configuration gateway in Dremio 11.0.0 is running.
  
• Dremio 11.0.0+ stops the AWS configuration gateway after you create a custom project or open an existing project.

Generating Project ID and an S3 Bucket Name

Dremio uses AWS tags to group your resources in a custom project. Use the projectInput endpoint to generate a project ID and a name for your S3 bucket before you create resources in AWS for your custom project.

Syntax

The endpoint requires one query parameter, the EC2 instance ID for your Dremio AWS Edition deployment.

Query parameter syntax

/aws/gateway/projectInput?instanceId={EC2-instance-id}

Example Request

Example requst

http://54.202.16.250:9047/aws/gateway/projectInput?instanceId=i-0941ec036ca73c48b

Example Response

Example response

{
  "statusCode": 200,
  "error": null,
  "id": "45a3cbf7-936a-4e4a-861d-17b5875a4fb2",
  "bucketName": "dremio-me-45a3cbf7-936a-4e4a-861d-17b5875a4fb2-08e9a0a2f9a8400f"
}

The id parameter in the response provides the ID for your custom Dremio project. The name of the S3 bucket in your custom project must match the value of the bucketName field.

In the next step, you will add this ID to the resources that you want to include in your custom project.

Creating and Tagging Project Resources

Dremio AWS Edition requires the following resource tags by resource type.

S3 Buckets

Tag Name	Tag Value
dremio_managed	true
dremio_project_id	{project-id-from- projectInput-endpoint}

EBS Volumes

Tag Name	Tag Value
dremio_managed	true
dremio_project_id	{project-id-from- projectInput-endpoint}
dremio_project_name	{user-defined-string}
dremio_project_pending	true

EFS Volumes

Tag Name	Tag Value
dremio_managed	true
dremio_project_id	{project-id-from- projectInput-endpoint}

Creating a Custom Project

Create the custom Dremio project using the customProject endpoint of the AWS configuration gateway.

Syntax

The customProject endpoint requires the following parameters in the HTTP request body:

Parameter	Data Type	Description
name	String	AWS Project Name
id	String	Custom AWS Project ID
engineSize	String	One of the following values: SMALL, MEDIUM, LARGE, XX_LARGE, XXX_LARGE, and CUSTOM
enableAutoBackups	Boolean	true or false
engineNodeType	String	One of the following values: STANDARD, HIGH_MEMORY, HIGH_CPU, HIGH_CAPACITY, and CUSTOM
iamInstanceProfile	String	Must be an empty string: ""
instanceId	String	The EC2 instance ID that you passed to the aws/gateway/projectInput endpoint
disablePublicIp	Boolean	true or false
ebsVolumeId	String	AWS identifier for an EBS volume to include in the custom project
efsVolumeName	String	AWS identifier for an EFS volume to include in the custom project
s3BucketName	String	AWS identifier for an S3 bucket to include in the custom project

Example Request

The following example creates a custom project with one EBS volume, one EFS volume, and one S3 bucket:

Example request

{
    "name": "MyProject",
    "id": "45a3cbf7-936a-4e4a-861d-17b5875a4fb2",
    "engineSize": "SMALL",
    "customNodeCount": 0,
    "enableAutoBackups": false,
    "engineNodeType": "STANDARD",
    "iamInstanceProfile": "",
    "instanceId": "i-0941ec036ca73c48b",
    "disablePublicIp": true,
    "ebsVolumeId": "vol-0f8064167d3d6cb98",
    "efsVolumeName": "fs-5a522a5f",
    "s3BucketName": "dremio-me-45a3cbf7-936a-4e4a-861d-17b5875a4fb2-08e9a0a2f9a8400f"
}

Example Response

The following is a successful response:

Example response

{
"statusCode": 200,
"error": null,
"instanceId": "i-0c749fab209c8fc68",
"id": "ba9277bb-41ea-4374-a64f-3b67c862de6d",
"name": "Custom1",
"iamInstanceProfile": "",
"engineNodeType": "STANDARD",
"engineSize": "MEDIUM",
"customNodeCount": 0,
"customEngineType": null,
"disablePublicIp": true,
"enableAutoBackups": true,
"ebsVolumeId": "vol-09c318e4b4a86975e",
"efsVolumeName": "fs-9e145c9b",
"s3BucketName": "dremio-me-ba9277bb-41ea-4374-a64f-3b67c862de6d-3c3a5327b8753a81"
}