Engines

An engine represents a Dremio Cloud entity that manages compute resources. Each engine has one or more replicas that are created for executing queries. An engine replica is a logical group of executors defined by the engine size. This API allows you to list existing engines and add, modify, enable/disable, and delete engines.

Engines Object

    "id": "e1cb20ff-a234-46f1-9f97-550e2bb5310f",
    "name": "firstEngine",
    "size": "XX_SMALL_V1",
    "activeReplicas": 0,
    "minReplicas": 0,
    "maxReplicas": 10,
    "autoStopDelaySeconds": 300,
    "queueTimeLimitSeconds": 300,
    "runtimeLimitSeconds": 0,
    "drainTimeLimitSeconds": 0,
    "state": "ENABLED",
    "cloudTags": [],
    "queriedAt": "Thu Jan 01 00:00:00 UTC 2021",
    "statusChangedAt": "Thu Aug 19 14:39:40 UTC 2021",
    "description": "",
    "maxConcurrency": 4,
    "instanceFamily": "M5D",
    "additionalEngineStateInfo": "NONE"

Engines Attributes

id String (UUID)

Unique identifier of an engine.

Example: e1cb20ff-a234-46f1-9f97-550e2bb5310f

name String

The user-defined name for an engine.

Example: firstEngine

size String

The size of the engine. Each engine replica that is created will be of this size.

Enum: XX_SMALL_V1, X_SMALL_V1, SMALL_V1, MEDIUM_V1, LARGE_V1, X_LARGE_V1, XX_LARGE_V1, XXX_LARGE_V1

Example: SMALL_V1

activeReplicas Integer

The number of engine replicas that are currently active.

Example: 0

minReplicas Integer

The minimum number of engine replicas that will be enabled at any given time.

Example: 0

maxReplicas Integer

The maximum number of engine replicas that will be enabled at any given time.

Example: 10

autoStopDelaySeconds Integer

The time (in seconds) that auto stop is delayed.

Example: 300

queueTimeLimitSeconds Integer

The maximum amount of time (in seconds) that a query will wait in the engine’s queue before it is canceled. The typical time to start a new replica is two minutes (120 seconds), so we recommend setting this to more than 120 seconds. Changing this setting does not affect queries that are either currently running or already in the queue.

Example: 300

runtimeLimitSeconds Integer

The maximum amount of time (in seconds) that a query can run before being terminated.

Example: 0

drainTimeLimitSeconds Integer

The maximum amount of time (in seconds) that an engine replica will continue to run after the engine is resized, disabled, or deleted before it is terminated and the running queries terminate.

Example: 0

state String

The current state of the engine.

Enum: DELETING, DISABLED, DISABLING, ENABLED, ENABLING, INVALID

Example: ENABLED

cloudTags Array of Object

Objects that list the keys and values for AWS or Azure tags.

Example: [{"key": "dremio", "value": "test"}]

queriedAt String

The date and time that the engine was last used to execute a query.

Example: Thu Jan 01 00:00:00 UTC 2021

statusChangedAt String

The date and time (in UTC time) that the state of the engine changed.

Example: Thu Aug 19 14:39:40 UTC 2021

description String

The description for the engine.

Example: engine for sales workloads

maxConcurrency Integer

The maximum number of concurrent queries that an engine replica can run.

Example: 4

instanceFamily String

Enum: M5D, M6ID, DDV4, DDV5

Example: M5D

M5D and M6ID are valid for engines on AWS clouds. DDV4 and DDV5 are valid for engines on Azure clouds.

additionalEngineStateInfo String

Not used. Has the value NONE.

Attributes of Objects in the `cloudTags` Array

key String

The key identifier for the tag.

Example: costCenter

value String

The value of the tag.

Example: dremio-team

Listing All Engines

Returns a list of all the engines in the specified project and the metadata for each engine.

Method and URL

GET /v0/projects/{project-id}/engines

Parameters

project-id Path String (UUID)

Unique identifier of the project you want to list engines for.

Example: 02d36975-73eb-47ed-9bb5-de73060380f6

Example Request

curl -X GET 'https://api.dremio.cloud/v0/projects/02d36975-73eb-47ed-9bb5-de73060380f6/engines' \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json'

Example Response

[
  {
    "id": "e1cb20ff-a234-46f1-9f97-550e2bb5310f",
    "name": "firstEngine",
    "size": "XX_SMALL_V1",
    "activeReplicas": 0,
    "minReplicas": 0,
    "maxReplicas": 10,
    "autoStopDelaySeconds": 300,
    "queueTimeLimitSeconds": 300,
    "runtimeLimitSeconds": 0,
    "drainTimeLimitSeconds": 0,
    "state": "ENABLED",
    "cloudTags": [],
    "queriedAt": "Thu Jan 01 00:00:00 UTC 2021",
    "statusChangedAt": "Thu Aug 19 14:39:40 UTC 2021",
    "description": "",
    "maxConcurrency": 4,
    "instanceFamily": "M5D",
    "additionalEngineStateInfo": "NONE"
  },
  {
    "id": "a99eb46a-8121-4b79-b2ed-917e07f96092",
    "name": "preview",
    "size": "XX_SMALL_V1",
    "activeReplicas": 0,
    "minReplicas": 0,
    "maxReplicas": 1,
    "autoStopDelaySeconds": 180,
    "queueTimeLimitSeconds": 300,
    "runtimeLimitSeconds": 0,
    "drainTimeLimitSeconds": 0,
    "state": "ENABLED",
    "cloudTags": [],
    "queriedAt": "Thu Jan 01 00:00:00 UTC 2021",
    "statusChangedAt": "Fri Aug 20 03:58:54 UTC 2021",
    "description": "",
    "maxConcurrency": 4,
    "instanceFamily": "M5D",
    "additionalEngineStateInfo": "NONE"
  }
]

Response Status Codes

200 OK

400 Bad Request

401 Unauthorized

Adding an Engine

Add an engine to the cloud account that is associated with the specified project.

Method and URL

POST /v0/projects/{project-id}/engines

Parameters

Engine

project-id Path String (UUID)

Unique identifier of the execution engine.

Example: c683b988-aa93-439c-a9ac-0d7f72b53d23

name Body String

The user-defined name for an engine.

Example: newengine

size Body String

The size of the engine that you want to create. Each engine replica that is created will be of this size.",

Enum: XX_SMALL_V1, X_SMALL_V1, SMALL_V1, MEDIUM_V1, LARGE_V1, X_LARGE_V1, XX_LARGE_V1, XXX_LARGE_V1

Example: XX_SMALL_V1

minReplicas Body Integer

The minimum number of engine replicas that will be enabled at any given time.

Example: 0

maxReplicas Body Integer

The maximum number of engine replicas that will be enabled at any given time.

Example: 1

autoStopDelaySeconds Body Integer

The time (in seconds) that auto stop is delayed.

Example: 7200

queueTimeLimitSeconds Body Integer

The maximum amount of time (in seconds) that a query will wait in the engine’s queue before it is canceled. This should not be set to less than two minutes (120 seconds), the typical time it takes to start a new replica. Changing this setting does not affect queries that are currently running or in the queue.

Example: 300

runtimeLimitSeconds Body Integer

The maximum amount of time (in seconds) that a query can run before being terminated.

Example: 0

drainTimeLimitSeconds Body Integer

The maximum amount of time (in seconds) that an engine replica will continue to run after the engine is resized, disabled, or deleted before it is terminated and the running queries terminate.

Example: 1800

cloudTags Body Array of Object Optional

Objects that list the keys and values for AWS or Azure tags.

description Body String Optional

The description for the engine.

Example: engine description

maxConcurrency Body Integer

The maximum number of concurrent queries that an engine replica can handle.

Example: 2

instanceFamily Body String Optional

Enum: M5D, M6ID, DDV4, DDV5

Example: M5D

M5D and M6ID are valid for engines on AWS clouds. DDV4 and DDV5 are valid for engines on Azure clouds.

requestId Body String (UUID)

The user-defined idempotency key, which is a unique value generated by the user that the server uses to register consequent retries of the same request. Generate the request ID by using a UUID generator tool. Read idempotent requests for more information.

Example: d9904076-01db-11ec-9a03-0242ac130003

Parameters of Objects in the `cloudTags` Array

key Body String Optional

The key identifier for the tag.

Example: costCenter

value Body String Optional

The value of the tag.

Example: dremio-team

Example Request

curl -X POST 'https://api.dremio.cloud/v0/projects/c683b988-aa93-439c-a9ac-0d7f72b53d23/engines' \
-H 'Authorization: Bearer <personal access token>' \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-d '{
  "name": "newengine",
  "size": "XX_SMALL_V1",
  "minReplicas": 0,
  "maxReplicas": 1,
  "autoStopDelaySeconds": 7200,
  "queueTimeLimitSeconds": 300,
  "runtimeLimitSeconds": 0,
  "drainTimeLimitSeconds": 1800,
  "cloudTags": [
    {
      "key": "tag1",
      "value": "value1"
    }
  ],
  "description": "engine description",
  "maxConcurrency": 2,
  "instanceFamily" : "M5D", 
  "requestId": "d9904076-01db-11ec-9a03-0242ac130003"
}'

Example Response

{
  "id": "168acd56-16a0-47df-91f8-5487a8fccfc4"
}

Response Status Codes

200 OK

400 Bad Request

401 Unauthorized

405 Method Not Allowed

500 Internal Server Error

Retrieving an Engine

Get the metadata for an existing engine by providing the project and engine IDs.

Method and URL

GET /v0/projects/{project-id}/engines/{id}

Parameters

project-id Path String (UUID)

Unique identifier of the execution engine.

Example: c683b988-aa93-439c-a9ac-0d7f72b53d23

id Path String (UUID)

The ID of the engine for which you want to retrieve metadata.

Example: 168acd56-16a0-47df-91f8-5487a8fccfc4

Example Request

curl -X GET 'https://api.dremio.cloud/v0/projects/c683b988-aa93-439c-a9ac-0d7f72b53d23/engines/168acd56-16a0-47df-91f8-5487a8fccfc4' \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json'

Example Response

{
  "id": "168acd56-16a0-47df-91f8-5487a8fccfc4",
  "name": "newengine",
  "size": "XX_SMALL_V1",
  "activeReplicas": 0,
  "minReplicas": 0,
  "maxReplicas": 1,
  "autoStopDelaySeconds": 7200,
  "queueTimeLimitSeconds": 300,
  "runtimeLimitSeconds": 0,
  "drainTimeLimitSeconds": 1800,
  "state": "ENABLED",
  "cloudTags": [
    {
      "key": "tag1",
      "value": "value1"
    }
  ],
  "queriedAt": "Thu Jan 01 00:00:00 UTC 2021",
  "statusChangedAt": "Fri Aug 20 17:29:18 UTC 2021",
  "description": "engine description",
  "maxConcurrency": 2,
  "instanceFamily": "M5D",
  "additionalEngineStateInfo": "NONE"
}

Response Status Codes

200 OK

400 Bad Request

401 Unauthorized

500 Internal Server Error

Modifying an Engine

Modify the engine by providing the project and engine IDs. Parameters that are not provided will be left unchanged. The engine name cannot be modified.

Method and URL

PUT /v0/projects/{project-id}/engines/{id}

Parameters

Engine

project-id Path String (UUID)

Unique identifier of the execution engine.

Example: c683b988-aa93-439c-a9ac-0d7f72b53d23

id Path String (UUID)

The ID of the engine for which you want to retrieve metadata.

Example: 168acd56-16a0-47df-91f8-5487a8fccfc4

size Body String Optional

The size of the engine that you want to change to. Each engine replica that is modified will be of this size.

Enum: XX_SMALL_V1, X_SMALL_V1, SMALL_V1, MEDIUM_V1, LARGE_V1, X_LARGE_V1, XX_LARGE_V1, XXX_LARGE_V1

Example: XX_SMALL_V1

minReplicas Body Integer Optional

The minimum number of engine replicas that will be enabled at any given time.

Example: 0

maxReplicas Body Integer Optional

The maximum number of engine replicas that will be enabled at any given time.

Example: 1

autoStopDelaySeconds Body Integer Optional

The time (in seconds) that auto stop is delayed.

Example: 7200

queueTimeLimitSeconds Body Integer Optional

Example: 300

runtimeLimitSeconds Body Integer Optional

The maximum amount of time (in seconds) that a query can run before being terminated.

Example: 0

drainTimeLimitSeconds Body Integer Optional

The maximum amount of time (in seconds) that an engine replica will continue to run after the engine is resized, disabled, or deleted before it is terminated and the running queries terminate.

Example: 1800

cloudTags Body String Optional

Objects that list the keys and values for AWS or Azure tags.

description Body String Optional

The description for the engine you are modifying.

Example: modified engine description

maxConcurrency Body Integer Optional

The maximum number of concurrent queries that an engine replica can run.

Example: 2

instanceFamily Body String Optional

Enum: M5D, M6ID, DDV4, DDV5

Example: M5D

M5D and M6ID are valid for engines on AWS clouds. DDV4 and DDV5 are valid for engines on Azure clouds.

Parameters of Objects in the `cloudTags` Array

key Body String Optional

The key identifier for the tag.

Example: costCenter

value Body String Optional

The value of the tag.

Example: dremio-team

Example Request

curl -X PUT 'https://api.dremio.cloud/v0/projects/c683b988-aa93-439c-a9ac-0d7f72b53d23/engines/168acd56-16a0-47df-91f8-5487a8fccfc4' \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json' \
-d '{
      "description": "modified engine description",
      "size": "XX_SMALL_V1",
      "autoStopDelaySeconds": 7200,
      "drainTimeLimitSeconds": 1800,
      "maxConcurrency": 2,
      "instanceFamily": "M5D", 
      "maxReplicas": 1,
      "minReplicas": 0,
      "queueTimeLimitSeconds": 300,
      "runtimeLimitSeconds": 0,
      "cloudTags":  [ ]
      }'

Example Response

No response

Response Status Codes

204 No Content

400 Bad Request

401 Unauthorized

500 Internal Server Error

Enabling an Engine

Enable an engine.

Method and URL

PUT /v0/projects/{project-id}/engines/{id}/enable

Parameters

project-id Path String (UUID)

Unique identifier of the execution engine.

Example: c683b988-aa93-439c-a9ac-0d7f72b53d23

id Path String (UUID)

The ID of the engine that you want to enable.

Example: 168acd56-16a0-47df-91f8-5487a8fccfc4

Example Request

curl -X PUT 'https://api.dremio.cloud/v0/projects/c683b988-aa93-439c-a9ac-0d7f72b53d23/engines/168acd56-16a0-47df-91f8-5487a8fccfc4/enable' \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json'

Example Response

No response

Response Status Codes

204 No Content

401 Unauthorized

500 Internal Server Error

Disabling an Engine

Disable an engine.

Method and URL

PUT /v0/projects/{project-id}/engines/{id}/disable

Parameters

project-id Path String (UUID)

Unique identifier of the execution engine.

Example: c683b988-aa93-439c-a9ac-0d7f72b53d23

id Path String (UUID)

The ID of the engine that you want to disable.

Example: 168acd56-16a0-47df-91f8-5487a8fccfc4

Example Request

curl -X PUT 'https://api.dremio.cloud/v0/projects/c683b988-aa93-439c-a9ac-0d7f72b53d23/engines/168acd56-16a0-47df-91f8-5487a8fccfc4/disable' \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json'

Example Response

No response

Response Status Codes

204 No Content

401 Unauthorized

500 Internal Server Error

Deleting an Engine

Delete an engine by providing the project and engine ID.

Method and URL

DELETE /v0/projects/{project-id}/engines/{id}

Parameters

project-id Path String (UUID)

Unique identifier of the execution engine.

Example: c683b988-aa93-439c-a9ac-0d7f72b53d23

id Path String (UUID)

The ID of the engine that you want to delete.

Example: 168acd56-16a0-47df-91f8-5487a8fccfc4

Example Request

curl -X DELETE 'https://api.dremio.cloud/v0/projects/c683b988-aa93-439c-a9ac-0d7f72b53d23/engines/168acd56-16a0-47df-91f8-5487a8fccfc4' \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json'

Example Response

No response

Response Status Codes

204 No Content

401 Unauthorized

500 Internal Server Error

Engines Attributes​

Attributes of Objects in the cloudTags Array​

Listing All Engines​

Parameters​

Response Status Codes​

Adding an Engine​

Parameters​

Engine​

Parameters of Objects in the cloudTags Array​

Response Status Codes​

Retrieving an Engine​

Parameters​

Response Status Codes​

Modifying an Engine​

Parameters​

Engine​

Parameters of Objects in the cloudTags Array​

Response Status Codes​

Enabling an Engine​

Parameters​

Response Status Codes​

Disabling an Engine​

Parameters​

Response Status Codes​

Deleting an Engine​

Parameters​

Response Status Codes​

Engines Attributes

Attributes of Objects in the `cloudTags` Array

Listing All Engines

Parameters

Response Status Codes

Adding an Engine

Parameters

Engine

Parameters of Objects in the `cloudTags` Array

Response Status Codes

Retrieving an Engine

Parameters

Response Status Codes

Modifying an Engine

Parameters

Engine

Parameters of Objects in the `cloudTags` Array

Response Status Codes

Enabling an Engine

Parameters

Response Status Codes

Disabling an Engine

Parameters

Response Status Codes

Deleting an Engine

Parameters

Response Status Codes