On this page

    Creating Custom Dremio Projects

    Dremio 11.0.0+ allows Dremio administrators to include their own S3 bucket, EBS volume, and EFS volume in a custom Dremio project, rather than using Dremio-provided default resources. Dremio 11.0.0+ provides an AWS configuration gateway with a REST API to create a project ID and a consistent set of AWS resource names that Dremio uses to group the resources, as well as a REST API to create the custom project with those resource inputs.

    To create a Dremio custom project in AWS Edition:

    1. Generate a project ID and a name for an S3 bucket using the /aws/gateway/projectInput endpoint on your Dremio AWS Edition deployment.
    2. Create and tag the AWS resources that you want to include in your custom Dremio project.
    3. Create the custom project with the /aws/gateway/customProject endpoint on your Dremio AWS Edition deployment.

    note:

    • A single Dremio 11.0.0+ AWS Edition deployment may create only one custom Dremio project.

    • A Dremio custom project is available only while the AWS configuration gateway in Dremio 11.0.0 is running.

    • Dremio 11.0.0+ stops the AWS configuration gateway after you create a custom project or open an existing project.

    ## Generating Project ID and an S3 Bucket Name Dremio uses AWS tags to group your resources in a custom project. Use the `projectInput` endpoint to generate a project ID and a name for your S3 bucket before you create resources in AWS for your custom project.

    Syntax

    The endpoint requires one query parameter, the EC2 instance ID for your Dremio AWS Edition deployment.

    /aws/gateway/projectInput?instanceId={EC2-instance-id}
    

    Example Request

    http://54.202.16.250:9047/aws/gateway/projectInput?instanceId=i-0941ec036ca73c48b
    

    Example Response

    {
      "statusCode": 200,
      "error": null,
      "id": "45a3cbf7-936a-4e4a-861d-17b5875a4fb2",
      "bucketName": "dremio-me-45a3cbf7-936a-4e4a-861d-17b5875a4fb2-08e9a0a2f9a8400f"
    }
    

    The id parameter in the response provides the ID for your custom Dremio project. The name of the S3 bucket in your custom project must match the value of the bucketName field.

    In the next step, you will add this ID to the resources that you want to include in your custom project.

    Creating and Tagging Project Resources

    Dremio AWS Edition requires the following resource tags by resource type.

    S3 Buckets

    Tag NameTag Value
    dremio_managedtrue
    dremio_project_id{project-id-from- projectInput-endpoint}

    EBS Volumes

    Tag NameTag Value
    dremio_managedtrue
    dremio_project_id{project-id-from- projectInput-endpoint}
    dremio_project_name{user-defined-string}
    dremio_project_pendingtrue

    EFS Volumes

    Tag NameTag Value
    dremio_managedtrue
    dremio_project_id{project-id-from- projectInput-endpoint}

    Creating a Custom Project

    Create the custom Dremio project using the customProject endpoint of the AWS configuration gateway.

    Syntax

    The customProject endpoint requires the following parameters in the HTTP request body:

    ParameterData TypeDescription
    nameStringAWS Project Name
    idStringCustom AWS Project ID
    engineSizeStringOne of the following values: SMALL, MEDIUM, LARGE, XX_LARGE, XXX_LARGE, and CUSTOM
    enableAutoBackupsBooleantrue or false
    engineNodeTypeStringOne of the following values: STANDARD, HIGH_MEMORY, HIGH_CPU, HIGH_CAPACITY, and CUSTOM
    iamInstanceProfileStringMust be an empty string: ""
    instanceIdStringThe EC2 instance ID that you passed to the aws/gateway/projectInput endpoint
    disablePublicIpBooleantrue or false
    ebsVolumeIdStringAWS identifier for an EBS volume to include in the custom project
    efsVolumeNameStringAWS identifier for an EFS volume to include in the custom project
    s3BucketNameStringAWS identifier for an S3 bucket to include in the custom project

    Example Request

    The following example creates a custom project with one EBS volume, one EFS volume, and one S3 bucket:

    {
    	"name": "MyProject",
    	"id": "45a3cbf7-936a-4e4a-861d-17b5875a4fb2",
    	"engineSize": "SMALL",
    	"customNodeCount": 0,
    	"enableAutoBackups": false,
    	"engineNodeType": "STANDARD",
    	"iamInstanceProfile": "",
    	"instanceId": "i-0941ec036ca73c48b",
    	"disablePublicIp": true,
    	"ebsVolumeId": "vol-0f8064167d3d6cb98",
    	"efsVolumeName": "fs-5a522a5f",
    	"s3BucketName": "dremio-me-45a3cbf7-936a-4e4a-861d-17b5875a4fb2-08e9a0a2f9a8400f"
    }
    

    Example Response

    The following is a successful response:

    {
    "statusCode": 200,
    "error": null,
    "instanceId": "i-0c749fab209c8fc68",
    "id": "ba9277bb-41ea-4374-a64f-3b67c862de6d",
    "name": "Custom1",
    "iamInstanceProfile": "",
    "engineNodeType": "STANDARD",
    "engineSize": "MEDIUM",
    "customNodeCount": 0,
    "customEngineType": null,
    "disablePublicIp": true,
    "enableAutoBackups": true,
    "ebsVolumeId": "vol-09c318e4b4a86975e",
    "efsVolumeName": "fs-9e145c9b",
    "s3BucketName": "dremio-me-ba9277bb-41ea-4374-a64f-3b67c862de6d-3c3a5327b8753a81"
    }