Version: current [26.x Preview]

Engine Management

Use the Engine Management API to manage your engines in Dremio:

Engine Management - Covers the objects and operations for managing engines.
Engine Configuration - Covers the objects and operations for engine configuration options.

Engine Object

{
  "id": "c3be6992-07d9-415d-89e4-571ceab8c51f",
  "name": "my-engine",
  "version": "xNMoJiZAyfA=",
  "spec": {
    "size": "2XSmall",
    "resourceAllocationOffset": "reserve-2-4",
    "targetCpuCapacity": "32C",
    "spillStorageSize": "128GB",
    "c3StorageSize": "128GB",
    "idleTimeout": "PT1H",
    "executorPodMetadata": {
      "labels": {
        "environment": "production",
        "team": "analytics"
      },
      "annotations": {
        "key1": "value1",
        "key2": "value2"
      },
      "nodeSelectors": {
        "kubernetes.io/hostname": "node-1"
      },
      "tolerations": [
        {
          "key": "example-key",
          "operator": "Exists",
          "effect": "NoExecute",
          "tolerationSeconds": 3600
        }
      ]
    }
  },
  "status": {
    "state": "RUNNING",
    "message": "",
    "runningReplicas": 1,
    "startingReplicas": 0,
    "stoppingReplicas": 0,
    "runningExecutors": 1,
    "startingExecutors": 0,
    "stoppingExecutors": 0,
    "replicas": [
      {
        "id": "000",
        "executors": [
          {
            "name": "executor-1",
            "memory": 10.45,
            "cpu": 5.33,
            "status": "RUNNING",
            "details": "No issues detected"
          }
        ]
      }
    ]
  }
}

Engine Attributes

id String (UUID)

Unique identifier for the engine, generated by Dremio.

Example: d480b906-6a2d-486e-a1f6-a2e86452eab8

name String

Name of the engine.

The name of the engine must follow these rules:

Must start with a lowercase alphanumeric character ([a-z0-9]).
Must end with a lowercase alphanumeric character ([a-z0-9]).
Must contain only lowercase alphanumeric characters or a hyphen ([\-a-z0-9]).
The maximum length is 30 characters.

Example: my-engine

version String

A tag that identifies the version of the resource.
When modifying a resource, you will need to send this tag to the server. It will reject the modification request if the provided version does not match the server's version, as this indicates that the resource has been modified since it was last read.

Example: xNMoJiZAyfA=

spec Object

Information about the engine specification.

Example: { "c3StorageSize": "128GB", "executorPodMetadata": { "annotations": { "key1": "value1", "key2": "value2" }, "labels": { "environment": "production", "team": "analytics" }, "nodeSelectors": { "kubernetes.io/hostname": "node-1" }, "tolerations": [ { "key": "example-key", "operator": "Exists", "effect": "NoExecute", "tolerationSeconds": 3600 } ] }, "idleTimeout": "PT1H", "resourceAllocationOffset": "reserve-2-4", "size": "2XSmall", "spillStorageSize": "128GB", "targetCpuCapacity": "32C"},

status Object

Information about the engine status.

Example: { "state": "RUNNING", "message": "Engine is running", "runningReplicas": 2, "startingReplicas": 0, "stoppingReplicas": 0, "replicas": [ { "id": "001", "executors": [ { "name": "executor-1", "cpu": 5.33, "memory": 10.45, "status": "RUNNING", "details": "No issues detected" }, { "name": "executor-2", "cpu": 7.25, "memory": 12.67, "status": "RUNNING", "details": "Stable performance" } ] } ] }

Attributes of the `spec` Object

size String

T-Shirt size of the engine, which must be one of the values predefined in the Engine Configuration.

Example: 2XSmall

resourceAllocationOffset String

Resource allocation offset level of the engine, which must be one of the values predefined in the Engine Configuration.

Example: reserve-2-4

targetCpuCapacity String

CPU target of the engine, which must be one of the values predefined in the Engine Configuration.

Example: 32C

spillStorageSize String

Disk size for the spill volume, which must be one of the values predefined in the Engine Configuration.

Example: 128GB

c3StorageSize String

Disk size for the C3 cache, which must be one of the values predefined in the Engine Configuration.

Example: 128GB

idleTimeout String

Amount of time before an engine stops automatically. The engine will also start automatically.
If absent, the engine must be started/stopped manually. If present but null, the system default will be used.
Specified as an ISO-8601 duration.

Example: PT1H

executorPodMetadata Object

Information about the Kubernetes metadata that can be applied to an executor pod.

Example: { "labels": { "environment": "production", "team": "analytics" }, "annotations": { "key1": "value1", "key2": "value2" }, "nodeSelectors": { "kubernetes.io/hostname": "node-1" }, "tolerations": [ { "key": "example-key", "operator": "Exists", "effect": "NoExecute", "tolerationSeconds": 3600 } ] }

Attributes of the `executorPodMetadata` Object

labels Object

Kubernetes metadata labels to apply to executor pods.
A map of label keys (string) to label values (string).
For more information, refer to https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/.

Example: { "environment": "production", "team": "analytics" }

annotations Object

Kubernetes metadata annotations to apply to executor pods.
A map of annotation keys (string) to annotation values (string).
For more information, refer to https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/.

Example: { "key1": "value1", "key2": "value2" }

nodeSelectors Object

Kubernetes node selectors to apply to executor pods.
A map of label keys (string) to label values (string).
For more information, refer to https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/.

Example: { "kubernetes.io/hostname": "node-1" }

tolerations Array of Objects

Kubernetes tolerations to apply to executor pods.
For more information, refer to https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/.

Example: { "key": "example-key", "operator": "Exists", "effect": "NoExecute", "tolerationSeconds": 3600 }

Attributes of the `toleration` Object

key String

The toleration key.

Example: example-key

operator String

The toleration operator.

Enum: Exists, Equal

Example: Exists

effect String

The toleration effect.

Enum: NoSchedule, PreferNoSchedule, NoExecute

Example: NoExecute

tolerationSeconds Integer

The toleration seconds, to be used only with effect NoExecute.

Example: 3600

value Integer

The toleration value.
Must be empty if operator is equal to Exists.

Attributes of the `engineStatus` Object

state String

Current state of the engine.

Enum: STARTING, RUNNING, STOPPING, STOPPED, WARNING, FAILED

Example: STARTING

message String

Human readable explanation of why the current state was achieved, including error messaging.

Example: No replicas are running

runningReplicas Integer

Number of running replicas.

Example: 1

startingReplicas Integer

Number of replicas starting.

Example: 1

stoppingReplicas Integer

Number of replicas stopping.

Example: 0

runningExecutors Integer

Number of executors replicas.

Example: 1

startingExecutors Integer

Number of executors starting.

Example: 1

stoppingExecutors Integer

Number of executors stopping.

Example: 0

replicas Array of Object

Running replicas with resource usage.

Example: { "id": "000", "executors": [ { "name": "dremio-executor-my-engine-000-0.dremio-cluster-pod.dremio.svc.cluster.local", "memory": 0.11473152822395594, "cpu": 0.07142857142857142, "status": "RUNNING" } ] }

Attributes of Objects in the `replica` Array

id String

Unique identifier for the replica in the engine, generated by Dremio.

Example: 001

executors Array of Object

Running executors with resource usage.

Example: [ { "name": "executor-1", "cpu": 5.33, "memory": 10.45, "status": "RUNNING", "details": "No issues detected" }, { "name": "executor-2", "cpu": 7.25, "memory": 12.67, "status": "RUNNING", "details": "Stable performance" } ]

Attributes of Objects in the `executor` Array

name String

Name of node. It's the hostname of the Kubernetes pod.

Example: dremio-master-0.dremio-cluster-pod.prod.svc.cluster.local

cpu Double

Percentage of CPU used over the last second for the replica.

Example: 5.33

memory Double

Percentage of direct memory in use for the replica.

Example: 10.45

status String

Status of the executor.

Enum: RUNNING, FAILED, WARNING

Example: RUNNING

details String

Error message sent by a running executor.

Example: Invalid version

Creating an Engine

Create a new engine.

Method and URL

POST /api/v3/engines

Parameters

name Body String

Name of the engine.

The name of the engine must follow these rules:

Must start with a lowercase alphanumeric character ([a-z0-9]).
Must end with a lowercase alphanumeric character ([a-z0-9]).
Must contain only lowercase alphanumeric characters or a hyphen ([\-a-z0-9]).
The maximum length is 30 characters.

Example: my-engine

newEngineSpec Body Object

Represents a new engine specification, with the option to use system defaults.

Example: { "size": "2XSmall", "targetCpuCapacity": "16C", "spillStorageSize": "100GB", "c3StorageSize": "100GB", "idleTimeout": "PT2H" }

Parameters of the `newEngineSpec` Object

size String

T-shirt size of the engine.

Example: 2XSmall

targetCpuCapacity String

CPU target of the engine, which must be one of the values predefined in the Engine Configuration.

Example: 32C

spillStorageSize String

Disk size for the spill volume, which must be one of the values predefined in the Engine Configuration.

Example: 128GB

c3StorageSize String

Disk size for the C3 cache, which must be one of the values predefined in the Engine Configuration.

Example: 128GB

idleTimeout String

Amount of time before an engine stops automatically. Engine will also start automatically. If unset, engine must be started/stopped manually. Specified as an ISO-8601 duration.

Example: PT1H

executorPodMetadata Object

Information about the Kubernetes metadata that can be applied to an executor pod.

Example Request

curl -X POST 'https://{hostname}/api/v3/engines' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer <PersonalAccessToken> \
--data-raw {
    "name": "my-engine",
    "spec": {
      "size": "2XSmall",
      "targetCpuCapacity": "16C",
      "spillStorageSize": "100GB",
      "c3StorageSize": "100GB",
      "idleTimeout": "PT2H"
    }
  }'

Example Response

{
  "id": "c3be6992-07d9-415d-89e4-571ceab8c51f",
  "name": "my-engine",
  "version": "FHj7/8hHG9U=",
  "spec": {
    "size": "2XSmall",
    "resourceAllocationOffset": "reserve-2-8",
    "targetCpuCapacity": "16C",
    "spillStorageSize": "100GB",
    "c3StorageSize": "100GB",
    "idleTimeout": "PT2H"
  },
  "status": {
    "state": "STARTING",
    "message": "",
    "runningReplicas": 0,
    "startingReplicas": 0,
    "stoppingReplicas": 0,
    "runningExecutors": 0,
    "startingExecutors": 0,
    "stoppingExecutors": 0,
    "replicas": []
  }
}

Response Status Codes

201 Engine created successfully

400 Bad Request

401 Unauthorized

403 Forbidden

404 Not Found

Listing Engines

Retrieve the list of engines.

Method and URL

GET /api/v3/engines

Example Request

curl -X GET 'https://{hostname}/api/v3/engines' \
--header 'Authorization: Bearer <PersonalAccessToken>'

Example Response

{
  "data": [
    {
      "id": "c3be6992-07d9-415d-89e4-571ceab8c51f",
      "name": "my-engine",
      "version": "reg6Ll6mCH8=",
      "spec": {
        "size": "2XSmall",
        "resourceAllocationOffset": "reserve-2-8",
        "targetCpuCapacity": "16C",
        "spillStorageSize": "100GB",
        "c3StorageSize": "100GB",
        "idleTimeout": "PT2H"
      },
      "status": {
        "state": "RUNNING",
        "message": "",
        "runningReplicas": 1,
        "startingReplicas": 0,
        "stoppingReplicas": 0,
        "runningExecutors": 1,
        "startingExecutors": 0,
        "stoppingExecutors": 0,
        "replicas": [
          {
            "id": "000",
            "executors": [
              {
                "name": "dremio-executor-my-engine-000-0.dremio-cluster-pod.dremio.svc.cluster.local",
                "memory": 0.25926326023549745,
                "cpu": 0.0,
                "status": "RUNNING"
              }
            ]
          }
        ]
      }
    }
  ]
}

Response Status Codes

200 List of engines

400 Bad Request

401 Unauthorized

403 Forbidden

404 Not Found

Retrieving an Engine by ID

Retrieve an engine by specifying the engines's ID.

Method and URL

GET /api/v3/engines/{id}

Parameters

id Path String (UUID)

Unique identifier of the engine that you want to retrieve.

Example: ec3fc60f-3b8a-4a26-bdaf-db97ae39c50d

Example Request

curl -X GET 'https://{hostname}/api/v3/engines/c3be6992-07d9-415d-89e4-571ceab8c51f' \
--header 'Authorization: Bearer <PersonalAccessToken>'

Example Response

{
  "id": "c3be6992-07d9-415d-89e4-571ceab8c51f",
  "name": "my-engine",
  "version": "reg6Ll6mCH8=",
  "spec": {
    "size": "2XSmall",
    "resourceAllocationOffset": "reserve-2-8",
    "targetCpuCapacity": "16C",
    "spillStorageSize": "100GB",
    "c3StorageSize": "100GB",
    "idleTimeout": "PT2H"
  },
  "status": {
    "state": "RUNNING",
    "message": "",
    "runningReplicas": 1,
    "startingReplicas": 0,
    "stoppingReplicas": 0,
    "runningExecutors": 1,
    "startingExecutors": 0,
    "stoppingExecutors": 0,
    "replicas": [
      {
        "id": "000",
        "executors": [
          {
            "name": "dremio-executor-my-engine-000-0.dremio-cluster-pod.dremio.svc.cluster.local",
            "memory": 0.25926326023549745,
            "cpu": 0.0,
            "status": "RUNNING"
          }
        ]
      }
    ]
  }
}

Response Status Codes

200 OK

400 Bad Request

401 Unauthorized

403 Forbidden

Updating an Engine

Update the specified engine.

Method and URL

PUT /api/v3/engines/{id}

Parameters

id Path String (UUID)

Unique identifier of the engine that you want to update.

Example: ec3fc60f-3b8a-4a26-bdaf-db97ae39c50d

version Body String

A tag that identifies the version of the resource that this modification is based on. You must obtain this tag by first getting the engine. The server will reject the request if the provided version does not match the server's version, as this indicates that the resource has been modified since it was last read.

Example: xNMoJiZAyfA=

spec Body Object

Represents an updated engine specification, with the option to use system defaults.

Example: { "size": "2XSmall", "targetCpuCapacity": "16C", "spillStorageSize": "100GB", "c3StorageSize": "100GB", "idleTimeout": "PT2H" }

Example Request

curl -X PUT 'https://{hostname}/api/v3/engines/c3be6992-07d9-415d-89e4-571ceab8c51f' \
--header 'Authorization: Bearer <PersonalAccessToken>' \
--header 'Content-Type: application/json' \
--data-raw ' {
    "name": "my-engine",
    "version": "neLlpVqsOdc=",
      "spec": {
        "size": "XSmall",
        "targetCpuCapacity": "16C",
        "spillStorageSize": "100GB",
        "c3StorageSize": "100GB",
        "idleTimeout": "PT1H"
      }
  }'

Example Response

{
  "id": "c3be6992-07d9-415d-89e4-571ceab8c51f",
  "name": "my-engine",
  "version": "ZLdaCiHCsic=",
  "spec": {
    "size": "XSmall",
    "resourceAllocationOffset": "reserve-2-8",
    "targetCpuCapacity": "16C",
    "spillStorageSize": "100GB",
    "c3StorageSize": "100GB",
    "idleTimeout": "PT1H"
  },
  "status": {
    "state": "STOPPING",
    "message": "",
    "runningReplicas": 0,
    "startingReplicas": 0,
    "stoppingReplicas": 0,
    "runningExecutors": 0,
    "startingExecutors": 0,
    "stoppingExecutors": 0,
    "replicas": []
  }
}

Response Status Codes

200 Engine updated successfully

400 Bad Request

401 Unauthorized

403 Forbidden

404 Engine not found

409 Version mismatch. Please fetch the latest resource and try again.

Stop an Engine

Stop the specified engine.

Method and URL

PUT /api/v3/engines/{id}/stop

Parameters

id Path String (UUID)

Unique identifier of the engine that you want to stop.

Example: ec3fc60f-3b8a-4a26-bdaf-db97ae39c50d

Example Request

curl -X PUT 'https://{hostname}/api/v3/engines/c3be6992-07d9-415d-89e4-571ceab8c51f/stop' \
--header 'Authorization: Bearer <PersonalAccessToken>'

Example Response

{
  "id": "c3be6992-07d9-415d-89e4-571ceab8c51f",
  "name": "my-engine",
  "version": "GdDM93EM8Tk=",
  "spec": {
    "size": "XSmall",
    "resourceAllocationOffset": "reserve-2-8",
    "targetCpuCapacity": "16C",
    "spillStorageSize": "100GB",
    "c3StorageSize": "100GB",
    "idleTimeout": "PT1H"
  },
  "status": {
    "state": "STOPPING",
    "message": "",
    "runningReplicas": 0,
    "startingReplicas": 0,
    "stoppingReplicas": 0,
    "runningExecutors": 0,
    "startingExecutors": 0,
    "stoppingExecutors": 0,
    "replicas": []
  }
}

Response Status Codes

200 Engine stopped successfully

401 Unauthorized

403 Forbidden

404 Engine not found

Start an Engine

Start the specified engine.

Method and URL

PUT /api/v3/engines/{id}/start

Parameters

id Path String (UUID)

Unique identifier of the engine that you want to start.

Example: ec3fc60f-3b8a-4a26-bdaf-db97ae39c50d

Example Request

curl -X PUT 'https://{hostname}/api/v3/engines/c3be6992-07d9-415d-89e4-571ceab8c51f/start' \
--header 'Authorization: Bearer <PersonalAccessToken>'

Example Response

{
  "id": "c3be6992-07d9-415d-89e4-571ceab8c51f",
  "name": "my-engine",
  "version": "aGnFL1ok1/Y=",
  "spec": {
    "size": "XSmall",
    "resourceAllocationOffset": "reserve-2-8",
    "targetCpuCapacity": "16C",
    "spillStorageSize": "100GB",
    "c3StorageSize": "100GB",
    "idleTimeout": "PT1H"
  },
  "status": {
    "state": "STARTING",
    "message": "",
    "runningReplicas": 0,
    "startingReplicas": 0,
    "stoppingReplicas": 0,
    "runningExecutors": 0,
    "startingExecutors": 0,
    "stoppingExecutors": 0,
    "replicas": []
  }
}

Response Status Codes

200 Engine started successfully

401 Unauthorized

403 Forbidden

404 Engine not found

Delete an Engine

Delete the specified engine.

Method and URL

DELETE /api/v3/engines/{id}

Parameters

id Path String (UUID)

Unique identifier of the engine that you want to start.

Example: ec3fc60f-3b8a-4a26-bdaf-db97ae39c50d

Example Request

curl -X DELETE 'https://{hostname}/api/v3/engines/c3be6992-07d9-415d-89e4-571ceab8c51f' \
--header 'Authorization: Bearer <PersonalAccessToken>'

Example Response

Response Status Codes

200 Engine deleted successfully

401 Unauthorized

403 Forbidden

404 Engine not found

Engine Attributes​

Attributes of the spec Object​

Attributes of the executorPodMetadata Object​

Attributes of the toleration Object​

Attributes of the engineStatus Object​

Attributes of Objects in the replica Array​

Attributes of Objects in the executor Array​

Creating an Engine​

Parameters​

Parameters of the newEngineSpec Object​

Response Status Codes​

Listing Engines​

Response Status Codes​

Retrieving an Engine by ID​

Parameters​

Response Status Codes​

Updating an Engine​

Parameters​

Response Status Codes​

Stop an Engine​

Parameters​

Response Status Codes​

Start an Engine​

Parameters​

Response Status Codes​

Delete an Engine​

Parameters​

Response Status Codes​

Engine Attributes

Attributes of the `spec` Object

Attributes of the `executorPodMetadata` Object

Attributes of the `toleration` Object

Attributes of the `engineStatus` Object

Attributes of Objects in the `replica` Array

Attributes of Objects in the `executor` Array

Creating an Engine

Parameters

Parameters of the `newEngineSpec` Object

Response Status Codes

Listing Engines

Response Status Codes

Retrieving an Engine by ID

Parameters

Response Status Codes

Updating an Engine

Parameters

Response Status Codes

Stop an Engine

Parameters

Response Status Codes

Start an Engine

Parameters

Response Status Codes

Delete an Engine

Parameters

Response Status Codes