Engine Configuration Options
Use the Engine Management API to manage your engines in Dremio:
- Engine Management - Covers the objects and operations for managing engines.
- Engine Configuration - Covers the objects and operations for engine configuration options.
{
"targetCpuCapacities": {
"capacities": [
{
"name": "16C",
"cpu": "16"
},
{
"name": "32C",
"cpu": "32"
}
],
"defaultCapacity": "32C"
},
"sizes": [
{
"name": "2XSmall",
"pods": 1,
"memory": "64Gi",
"cpuScaleFactor": 0.5
},
{
"name": "XSmall",
"pods": 1,
"memory": "128Gi",
"cpuScaleFactor": 1.0
},
{
"name": "Small",
"pods": 2,
"memory": "128Gi",
"cpuScaleFactor": 1.0
},
{
"name": "Medium",
"pods": 4,
"memory": "128Gi",
"cpuScaleFactor": 1.0
},
{
"name": "Large",
"pods": 8,
"memory": "128Gi",
"cpuScaleFactor": 1.0
},
{
"name": "XLarge",
"pods": 16,
"memory": "128Gi",
"cpuScaleFactor": 1.0
},
{
"name": "2XLarge",
"pods": 32,
"memory": "128Gi",
"cpuScaleFactor": 1.0
}
],
"resourceAllocationOffsets": {
"offsets": [
{
"name": "reserve-0-0",
"cpu": "0",
"memory": "0Gi",
"action": "Reserve"
},
{
"name": "reserve-2-4",
"cpu": "2",
"memory": "4Gi",
"action": "Reserve"
},
{
"name": "reserve-2-8",
"cpu": "2",
"memory": "8Gi",
"action": "Reserve"
},
{
"name": "reserve-2-16",
"cpu": "2",
"memory": "16Gi",
"action": "Reserve"
}
],
"defaultOffset": "reserve-2-8"
},
"storage": {
"spillStorageSizes": [
{
"name": "100GB",
"storage": "100Gi"
},
{
"name": "250GB",
"storage": "250Gi"
},
{
"name": "500GB",
"storage": "500Gi"
}
],
"defaultSpillStorageSize": "100GB",
"c3StorageSizes": [
{
"name": "100GB",
"storage": "100Gi"
},
{
"name": "250GB",
"storage": "250Gi"
},
{
"name": "500GB",
"storage": "500Gi"
}
],
"defaultC3StorageSize": "100GB"
},
"idleTimeouts": {
"durations": [
"PT2H",
"PT1H30M",
"PT1H",
"PT30M",
"PT15M",
"PT10M",
"PT5M"
],
"defaultDuration": "PT2H",
"maximumDuration": "PT12H"
}
}
Engine Configuration Options Attributes
targetCpuCapacities Object
Information about the possible CPU capacities optios to configure engines and which is the default one.
Example: { "capacities": [ { "name": "16C", "cpu": "16" }, { "name": "32C", "cpu": "32" } ], "defaultCapacity": "32C" }
sizes Array of Objects
Information about the possible sizes to configure engines.
Example: [ { "name": "2XSmall", "pods": 1, "memory": "64Gi", "cpuScaleFactor": 0.5 }, { "name": "XSmall","pods": 1, "memory": "128Gi", "cpuScaleFactor": 1.0 }, { "name": "Small", "pods": 2, "memory": "128Gi","cpuScaleFactor": 1.0 }, { "name": "Medium", "pods": 4, "memory": "128Gi", "cpuScaleFactor": 1.0 }, { "name": "Large", "pods": 8, "memory": "128Gi", "cpuScaleFactor": 1.0 }, { "name": "XLarge", "pods": 16, "memory": "128Gi", "cpuScaleFactor": 1.0 }, { "name": "2XLarge", "pods": 32, "memory": "128Gi", "cpuScaleFactor": 1.0 } ]
resourceAllocationOffsets Object
Information about the possible pod resource allocation options to configure engines and which is the default one.
Example: { "offsets": [ { "name": "reserve-0-0", "cpu": "0", "memory": "0Gi", "action": "Reserve" }, { "name": "reserve-2-4", "cpu": "2", "memory": "4Gi", "action": "Reserve" }, { "name": "reserve-2-8", "cpu": "2", "memory": "8Gi", "action": "Reserve" }, { "name": "reserve-2-16", "cpu": "2", "memory": "16Gi", "action": "Reserve" } ], "defaultOffset": "reserve-2-8" }
storage Object
Information all pod storage options.
Example: { "spillStorageSizes": [ { "name": "100GB", "storage": "100Gi" }, { "name": "250GB", "storage": "250Gi" }, { "name": "500GB", "storage": "500Gi" } ], "defaultSpillStorageSize": "100GB", "c3StorageSizes": [ { "name": "100GB", "storage": "100Gi" }, { "name": "250GB", "storage": "250Gi" }, { "name": "500GB", "storage": "500Gi" } ], "defaultC3StorageSize": "100GB" }
idleTimeouts Object
Information all idle timeout options, which is the default one, and the maximum duration.
Example: { "durations": [ "PT2H", "PT1H30M", "PT1H", "PT30M", "PT15M", "PT10M", "PT5M" ], "defaultDuration": "PT2H", "maximumDuration": "PT12H" }
Attributes of the targetCpuCapacities Object
capacities Array of Objects
The possible CPU capacities to configure engines.
Example: [ { "name": "16C", "cpu": "16" }, { "name": "32C", "cpu": "32" } ]
defaultCapacity String
If not defined for the engine, this the default capacity for it.
Example: 32C
Attributes of Objects in the capacities Array
name String
The name of the cpu capacity.
Example: 32C
cpu String
The unadjusted cpu allocated to each executor pod of each replica of an engine. Expressed as a Kubernetes Quantity.
Example: 32
Attributes of Objects in the sizes Array
name String
The name of the engine size, to be used as the key.
Example: 2XSmall
pods Integer
The number of executor pods created for each replica of an engine of this size.
Example: 8
memory String
The memory allocated to each executor pod of each replica of an engine of this size. Expressed as a Kubernetes Quantity.
Example: 64Gi
cpuScaleFactor Float
The scaling applied to the chosen TargetCpuCapacity for this engine size. Used to declare sizes that have smaller CPU allocations than those provided by Kubernetes nodes. The value must be between 0.0 and 1.0. Defaults to 1.0.
Example: 1.0
Attributes of the resourceAllocationOffsets Object
offsets Array of Objects
The possible resource allocation offset options to configure engines.
Example: { "name": "reserve-0-0", "cpu": "0", "memory": "0Gi", "action": "Reserve" }, { "name": "reserve-2-4", "cpu": "2", "memory": "4Gi", "action": "Reserve" }, { "name": "reserve-2-8", "cpu": "2", "memory": "8Gi", "action": "Reserve" }, { "name": "reserve-2-16", "cpu": "2", "memory": "16Gi", "action": "Reserve" }
defaultOffset String
The default offset used for engines where a specific value is not specified.
Example: reserve-2-8
Attributes of the Objects in the offsets Array
name String
The name of the resource allocation offset, to be used as key.
Example: reserve-2-8
cpu String
The CPU adjustment to be made to each executor pod of each replica of an engine with this capacity.
Expressed as a Kubernetes Quantity.
Example: 0
memory String
The memory adjustment to be made to each executor pod of each replica of an engine with this capacity. Expressed as a Kubernetes Quantity.
Example: 8Gi
action String
The offset type, where 'Reserve' subtracts the offset and 'OverCommit' adds it to the pod allocations.
Example: Reserve
Attributes of the storage Object
spillStorageSizes Array of Objects
The set of all spill storage size options.
Example: [ { "name": "100GB", "storage": "100Gi" }, { "name": "250GB", "storage": "250Gi" }, { "name": "500GB", "storage": "500Gi" } ]
defaultSpillStorageSize String
The default spill storage size to used for engines where a specific value is not specified.
Example: 100GB
c3StorageSizes Array of Objects
The set of all C3 storage size options.
Example: [ { "name": "100GB", "storage": "100Gi" }, { "name": "250GB", "storage": "250Gi" }, { "name": "500GB", "storage": "500Gi" } ]
defaultC3StorageSize String
The default C3 storage size to used for engines where a specific value is not specified.
Example: 100GB
Attributes of the Objects in the spillStorageSizes Array
name String
The name of the storage size, to be used as the key.
Example: 100GB
storage String
The storage allocated to each executor pod of each replica of an engine.
Expressed as a Kubernetes Quantity.
Example: 128Gi
Attributes of the Objects in the c3StorageSizes Array
name String
The name of the storage size, to be used as the key.
Example: 100GB
storage String
The storage allocated to each executor pod of each replica of an engine.
Expressed as a Kubernetes Quantity.
Example: 128Gi
Attributes of the idleTimeouts Object
durations Array of String
The set of all idle timeout options.
Example: [ "PT2H", "PT1H30M", "PT1H", "PT30M", "PT15M", "PT10M", "PT5M" ]
defaultDuration String
The default idle timeout to used for engines where a specific value is not specified.
Example: PT1H
maximumDuration String
The maximum idle timeout that can be specified by the user.
Example: PT12H
Retrieving the Configuration Options
Retrieve all engine configuration options.
Method and URLGET /api/v3/engines/configuration/options
curl -X POST 'https://{hostname}/api/v3/engines/configuration/options' \
--header 'Authorization: Bearer <PersonalAccessToken>
{
"targetCpuCapacities": {
"capacities": [
{
"name": "16C",
"cpu": "16"
},
{
"name": "32C",
"cpu": "32"
}
],
"defaultCapacity": "32C"
},
"sizes": [
{
"name": "2XSmall",
"pods": 1,
"memory": "64Gi",
"cpuScaleFactor": 0.5
},
{
"name": "XSmall",
"pods": 1,
"memory": "128Gi",
"cpuScaleFactor": 1.0
},
{
"name": "Small",
"pods": 2,
"memory": "128Gi",
"cpuScaleFactor": 1.0
},
{
"name": "Medium",
"pods": 4,
"memory": "128Gi",
"cpuScaleFactor": 1.0
},
{
"name": "Large",
"pods": 8,
"memory": "128Gi",
"cpuScaleFactor": 1.0
},
{
"name": "XLarge",
"pods": 16,
"memory": "128Gi",
"cpuScaleFactor": 1.0
},
{
"name": "2XLarge",
"pods": 32,
"memory": "128Gi",
"cpuScaleFactor": 1.0
}
],
"resourceAllocationOffsets": {
"offsets": [
{
"name": "reserve-0-0",
"cpu": "0",
"memory": "0Gi",
"action": "Reserve"
},
{
"name": "reserve-2-4",
"cpu": "2",
"memory": "4Gi",
"action": "Reserve"
},
{
"name": "reserve-2-8",
"cpu": "2",
"memory": "8Gi",
"action": "Reserve"
},
{
"name": "reserve-2-16",
"cpu": "2",
"memory": "16Gi",
"action": "Reserve"
}
],
"defaultOffset": "reserve-2-8"
},
"storage": {
"spillStorageSizes": [
{
"name": "100GB",
"storage": "100Gi"
},
{
"name": "250GB",
"storage": "250Gi"
},
{
"name": "500GB",
"storage": "500Gi"
}
],
"defaultSpillStorageSize": "100GB",
"c3StorageSizes": [
{
"name": "100GB",
"storage": "100Gi"
},
{
"name": "250GB",
"storage": "250Gi"
},
{
"name": "500GB",
"storage": "500Gi"
}
],
"defaultC3StorageSize": "100GB"
},
"idleTimeouts": {
"durations": [
"PT2H",
"PT1H30M",
"PT1H",
"PT30M",
"PT15M",
"PT10M",
"PT5M"
],
"defaultDuration": "PT2H",
"maximumDuration": "PT12H"
}
}
Response Status Codes
201 Default configuration
400 Bad Request
401 Unauthorized
403 Forbidden
404 Not Found
Retrieving the Deployment Type
Retrieve the deployment type.
Method and URLGET /api/v3/engines/configuration/deployment-type
curl -X GET 'https://{hostname}/api/v3/engines/configuration/deployment-type' \
--header 'Authorization: Bearer <PersonalAccessToken>'
{
"type": "KUBERNETES"
}
Response Status Codes
200 Supported cluster type
400 Bad Request
401 Unauthorized
403 Forbidden
404 Not Found