Queue Enterprise
Use the Workload Management (WLM) API to create, retrieve, update, and delete WLM queues.
Queue Object{
"data": [
{
"id": "1990e713-3cd2-458c-89e1-68995c2c1047",
"tag": "BNGRmgfEnDg=",
"name": "High Cost Reflections",
"maxMemoryPerNodeBytes": 8589934592,
"maxQueryMemoryPerNodeBytes": 8589934592,
"cpuTier": "BACKGROUND",
"maxAllowedRunningJobs": 10,
"maxStartTimeoutMs": 300000,
"maxRunTimeoutMs": 300000,
"engineId": "DATA"
},
{
"id": "0dbc50a0-034d-40f6-92f7-ff11eda0c760",
"tag": "HM2D9XElG3U=",
"name": "Low Cost Reflections",
"cpuTier": "BACKGROUND",
"maxAllowedRunningJobs": 10,
"maxStartTimeoutMs": 86400000
},
{
"id": "450ea2a5-9a64-4679-99cb-7b01bf6bba27",
"tag": "p22KaFcaB7g=",
"name": "COPY & OPTIMIZATION Queue",
"maxMemoryPerNodeBytes": 4294967296,
"maxQueryMemoryPerNodeBytes": 4294967296,
"cpuTier": "MEDIUM",
"maxAllowedRunningJobs": 2,
"maxStartTimeoutMs": 300000,
"engineId": "YARN"
},
{
"id": "a254d63e-9b0e-41be-af4a-1acc5bfe2332",
"tag": "//gNL3Ta2bY=",
"name": "Low Cost User Queries",
"cpuTier": "MEDIUM",
"maxAllowedRunningJobs": 100,
"maxStartTimeoutMs": 300000
},
{
"id": "c2917cce-b566-4c6a-be63-2e28488a6928",
"tag": "wa+vYmA73gU=",
"name": "High Cost User Queries",
"cpuTier": "MEDIUM",
"maxAllowedRunningJobs": 10,
"maxStartTimeoutMs": 300000
}
]
}
Queue Attributes
data Array of Object
List of queue objects in the Dremio instance.
Attributes of the data Object
id String (UUID)
Unique identifier of the queue, in UTC format.
Example: 1990e713-3cd2-458c-89e1-68995c2c1047
tag String
Unique identifier of the version of the queue. Dremio changes the tag whenever the queue changes and uses the tag to ensure that PUT requests apply to the most recent version of the queue.
Example: BNGRmgfEnDg=
name String
User-provided name for the queue.
Example: High Cost Reflections
maxMemoryPerNodeBytes Integer
Total memory (in bytes) that all queries running in parallel in a given queue can use per executor node.
maxQueryMemoryPerNodeBytes Integer
Total memory (in bytes) that each query in a given queue can use per executor node.
Example: 8589934592
cpuTier String
Amount of CPU time that threads get compared to other threads.
Enum: BACKGROUND, LOW, MEDIUM, HIGH, CRITICAL
Example: BACKGROUND
maxAllowedRunningJobs Integer
Number of queries that are allowed to run in parallel.
Example: 10
maxStartTimeoutMs Integer
Maximum length of time that a query can wait in the queue before it is cancelled, in milliseconds.
Example: 300000
maxRunTimeoutMs Integer
Maximum length of time that a query can run before it is cancelled, in milliseconds.
Example: 300000
engineId String
Name of the execution engine to which the queue's queries are routed. If you do not specify an engineId, the queue's queries run on any engine that is available at the time of execution. The engineID attribute is omitted from the queue object if no engine is specified. For more information, read Workload Management-based Routing.
Example: DATA
Creating a Queue
Create a WLM queue.
Method and URLPOST /api/v3/wlm/queue
Parameters
name Body String
User-provided name for the queue.
Example: High Cost Reflections
maxMemoryPerNodeBytes Body Integer Optional
Total memory (in bytes) that all queries running in parallel in a given queue can use per executor node.
Example: 8589934592
maxQueryMemoryPerNodeBytes Body Integer Optional
Total memory (in bytes) that each query in a given queue can use per executor node.
Example: 8589934592
cpuTier Body String Optional
Amount of CPU time that threads should get compared to other threads. Default is MEDIUM.
Enum: BACKGROUND, LOW, MEDIUM, HIGH, CRITICAL
Example: BACKGROUND
maxAllowedRunningJobs Body Integer Optional
Number of queries that are allowed to run in parallel.
Example: 10
maxStartTimeoutMs Body Integer Optional
Maximum length of time that a query can wait in the queue before it is cancelled, in milliseconds.
Example: 300000
maxRunTimeoutMs Body Integer Optional
Maximum length of time that a query can run before it is cancelled, in milliseconds.
Example: 300000
engineId Body String Optional
Name of the execution engine to which the queue's queries should be routed. If you do not specify an engineId, the queue's queries run on any engine that is available at the time of execution. For more information, read Workload Management-based Routing.
Example: DATA
Example Requestcurl -X POST 'https://{hostname}/api/v3/wlm/queue' \
--header 'Authorization: Bearer <PersonalAccessToken>' \
--header 'Content-Type: application/json' \
--data-raw '{
"name": "High Cost Reflections",
"maxQueryMemoryPerNodeBytes": 8589934592,
"maxMemoryPerNodeBytes": 8589934592,
"cpuTier": "BACKGROUND",
"maxAllowedRunningJobs": 10,
"maxStartTimeoutMs": 300000,
"maxRunTimeoutMs": 300000,
"engineId": "DATA"
}'
{
"id": "1990e713-3cd2-458c-89e1-68995c2c1047",
"tag": "BNGRmgfEnDg=",
"name": "High Cost Reflections",
"maxMemoryPerNodeBytes": 8589934592,
"maxQueryMemoryPerNodeBytes": 8589934592,
"cpuTier": "BACKGROUND",
"maxAllowedRunningJobs": 10,
"maxStartTimeoutMs": 300000,
"maxRunTimeoutMs": 300000,
"engineId": "DATA"
}
Response Status Codes
200 OK
400 Bad Request
401 Unauthorized
403 Forbidden
404 Not Found
405 Method Not Allowed
409 Conflict
Retrieving All Queues
Retrieve all WLM queues.
Method and URLGET /api/v3/wlm/queue
curl -X GET 'https://{hostname}/api/v3/wlm/queue' \
--header 'Authorization: Bearer <PersonalAccessToken>' \
--header 'Content-Type: application/json'
{
"data": [
{
"id": "1990e713-3cd2-458c-89e1-68995c2c1047",
"tag": "BNGRmgfEnDg=",
"name": "High Cost Reflections",
"maxMemoryPerNodeBytes": 8589934592,
"maxQueryMemoryPerNodeBytes": 8589934592,
"cpuTier": "BACKGROUND",
"maxAllowedRunningJobs": 10,
"maxStartTimeoutMs": 300000,
"maxRunTimeoutMs": 300000,
"engineId": "DATA"
},
{
"id": "0dbc50a0-034d-40f6-92f7-ff11eda0c760",
"tag": "HM2D9XElG3U=",
"name": "Low Cost Reflections",
"cpuTier": "BACKGROUND",
"maxAllowedRunningJobs": 10,
"maxStartTimeoutMs": 86400000
},
{
"id": "450ea2a5-9a64-4679-99cb-7b01bf6bba27",
"tag": "p22KaFcaB7g=",
"name": "COPY & OPTIMIZATION Queue",
"maxMemoryPerNodeBytes": 4294967296,
"maxQueryMemoryPerNodeBytes": 4294967296,
"cpuTier": "MEDIUM",
"maxAllowedRunningJobs": 2,
"maxStartTimeoutMs": 300000,
"engineId": "YARN"
},
{
"id": "a254d63e-9b0e-41be-af4a-1acc5bfe2332",
"tag": "//gNL3Ta2bY=",
"name": "Low Cost User Queries",
"cpuTier": "MEDIUM",
"maxAllowedRunningJobs": 100,
"maxStartTimeoutMs": 300000
},
{
"id": "c2917cce-b566-4c6a-be63-2e28488a6928",
"tag": "wa+vYmA73gU=",
"name": "High Cost User Queries",
"cpuTier": "MEDIUM",
"maxAllowedRunningJobs": 10,
"maxStartTimeoutMs": 300000
}
]
}
Response Status Codes
200 OK
401 Unauthorized
403 Forbidden
404 Not Found
500 Internal Server Error
Retrieving a Queue by ID
Retrieve a specific WLM queue by the queue's ID.
Method and URLGET /api/v3/wlm/queue/{id}
Parameters
id Path String (UUID)
Unique identifier of the queue you want to retrieve, in UTC format.
Example: 1990e713-3cd2-458c-89e1-68995c2c1047
Example Requestcurl -X GET 'https://{hostname}/api/v3/wlm/queue/1990e713-3cd2-458c-89e1-68995c2c1047' \
--header 'Authorization: Bearer <PersonalAccessToken>' \
--header 'Content-Type: application/json'
{
"id": "1990e713-3cd2-458c-89e1-68995c2c1047",
"tag": "BNGRmgfEnDg=",
"name": "High Cost Reflections",
"maxMemoryPerNodeBytes": 8589934592,
"maxQueryMemoryPerNodeBytes": 8589934592,
"cpuTier": "BACKGROUND",
"maxAllowedRunningJobs": 10,
"maxStartTimeoutMs": 300000,
"maxRunTimeoutMs": 300000,
"engineId": "DATA"
}
Response Status Codes
200 OK
401 Unauthorized
403 Forbidden
404 Not Found
500 Internal Server Error
Retrieving a Queue by Name
Retrieve a specific WLM queue by the queue's name.
Method and URLGET /api/v3/wlm/queue/by-name/{name}
Parameters
name Path String
Name for the queue you want to retrieve. If the queue name includes special characters for a URL, such as spaces, use URL encoding to replace the special characters with their UTF-8-equivalent characters. For example, "Dremio University" should be Dremio%20University in the URL path.
Example: High%20Cost%20Reflections
Example Requestcurl -X GET 'https://{hostname}/api/v3/wlm/queue/by-name/High%20Cost%20Reflections' \
--header 'Authorization: Bearer <PersonalAccessToken>' \
--header 'Content-Type: application/json'
{
"id": "1990e713-3cd2-458c-89e1-68995c2c1047",
"tag": "BNGRmgfEnDg=",
"name": "High Cost Reflections",
"maxMemoryPerNodeBytes": 8589934592,
"maxQueryMemoryPerNodeBytes": 8589934592,
"cpuTier": "BACKGROUND",
"maxAllowedRunningJobs": 10,
"maxStartTimeoutMs": 300000,
"maxRunTimeoutMs": 300000,
"engineId": "DATA"
}
Response Status Codes
200 OK
401 Unauthorized
403 Forbidden
404 Not Found
500 Internal Server Error
Updating a Queue
Update the specified WLM queue.
Method and URLPUT /api/v3/wlm/queue/{id}
Parameters
id Path String (UUID)
Unique identifier of the queue you want to update, in UTC format.
Example: 1990e713-3cd2-458c-89e1-68995c2c1047
tag Body String
Unique identifier of the version of the queue to update. Dremio uses the tag to ensure that you are updating the most recent version of the queue.
Example: BNGRmgfEnDg=
name Body String
User-provided name for the queue.
Example: High Cost Reflections
maxMemoryPerNodeBytes Body Integer Optional
Total memory (in bytes) that all queries running in parallel in a given queue can use per executor node. If you omit the maxMemoryPerNodeBytes parameter in a PUT request, Dremio removes the existing maxMemoryPerNodeBytes value from the queue. To keep the existing value while making other updates, include the existing maxMemoryPerNodeBytes parameter and value in the PUT request.
Example: 8589934592
maxQueryMemoryPerNodeBytes Body Integer Optional
Total memory (in bytes) that each query in a given queue can use per executor node. If you omit the maxQueryMemoryPerNodeBytes parameter in a PUT request, Dremio removes the existing maxQueryMemoryPerNodeBytes value from the queue. To keep the existing value while making other updates, include the existing maxQueryMemoryPerNodeBytes parameter and value in the PUT request.
Example: 8589934592
cpuTier Body String Optional
Amount of CPU time that threads should get compared to other threads. Default is MEDIUM. If you omit the cpuTier parameter in a PUT request, Dremio replaces it with the default value. To keep the existing setting while making other updates, include the existing cpuTier parameter and setting in the PUT request.
Enum: BACKGROUND, LOW, MEDIUM, HIGH, CRITICAL
Example: LOW
maxAllowedRunningJobs Body Integer Optional
Number of queries that are allowed to run in parallel. If you omit the maxAllowedRunningJobs parameter in a PUT request, Dremio removes the existing maxAllowedRunningJobs value from the queue. To keep the existing value while making other updates, include the existing maxAllowedRunningJobs parameter and value in the PUT request.
Example: 100
maxStartTimeoutMs Body Integer Optional
Maximum length of time that a query can wait in the queue before it is cancelled, in milliseconds. If you omit the maxStartTimeoutMs parameter in a PUT request, Dremio removes the existing maxStartTimeoutMs setting from the queue. To keep the existing setting while making other updates, include the existing maxStartTimeoutMs parameter and setting in the PUT request.
Example: 300000
maxRunTimeoutMs Body Integer Optional
Maximum length of time that a query is allowed to run before it is cancelled, in milliseconds. If you omit the maxRunTimeoutMs parameter in a PUT request, Dremio removes the existing maxRunTimeoutMs value from the queue. To keep the existing value while making other updates, include the existing maxRunTimeoutMs parameter and value in the PUT request.
Example: 300000
engineId Body String
Name of the execution engine to which the queue's queries should be routed. If you do not specify an engineId, the queue's queries run on any engine that is available at the time of execution. For more information, read Workload Management-based Routing.
Example: DATA
Example Requestcurl -X PUT 'https://{hostname}/api/v3/wlm/queue/1990e713-3cd2-458c-89e1-68995c2c1047' \
--header 'Authorization: Bearer <PersonalAccessToken>' \
--header 'Content-Type: application/json' \
--data-raw '{
"tag": "BNGRmgfEnDg=",
"name": "High Cost Reflections",
"maxMemoryPerNodeBytes": 8589934592,
"maxQueryMemoryPerNodeBytes": 8589934592,
"cpuTier": "LOW",
"maxAllowedRunningJobs": 100,
"maxStartTimeoutMs": 300000,
"maxRunTimeoutMs": 300000,
"engineId": "DATA"
}'
{
"id": "1990e713-3cd2-458c-89e1-68995c2c1047",
"tag": "xQh6KNyEjus=",
"name": "High Cost Reflections",
"maxMemoryPerNodeBytes": 8589934592,
"maxQueryMemoryPerNodeBytes": 8589934592,
"cpuTier": "LOW",
"maxAllowedRunningJobs": 100,
"maxStartTimeoutMs": 300000,
"maxRunTimeoutMs": 300000,
"engineId": "DATA"
}
Response Status Codes
200 OK
400 Bad Request
401 Unauthorized
403 Forbidden
404 Not Found
409 Conflict
500 Internal Server Error
Deleting a Queue
Delete the specified WLM queue.
Method and URLDELETE /api/v3/wlm/queue/{id}
Parameters
id Path String (UUID)
Unique identifier of the queue that you want to delete, in UTC format.
Example: 1990e713-3cd2-458c-89e1-68995c2c1047
Example Requestcurl -X DELETE 'https://{hostname}/api/v3/wlm/queue/1990e713-3cd2-458c-89e1-68995c2c1047' \
--header 'Authorization: Bearer <PersonalAccessToken>' \
--header 'Content-Type: application/json'
No response
Response Status Codes
200 OK
400 Bad Request
401 Unauthorized
403 Forbidden
404 Not Found
405 Method Not Allowed