Queue enterprise
Use the Workload Management (WLM) API to create, retrieve, update, and delete WLM queues.
Queue Object{
"data": [
{
"id": "1990e713-3cd2-458c-89e1-68995c2c1047",
"tag": "BNGRmgfEnDg=",
"name": "High Cost Reflections",
"maxMemoryPerNodeBytes": 8589934592,
"maxQueryMemoryPerNodeBytes": 8589934592,
"cpuTier": "BACKGROUND",
"maxAllowedRunningJobs": 10,
"maxStartTimeoutMs": 300000,
"maxRunTimeoutMs": 300000,
"engineId": "DATA"
},
{
"id": "0dbc50a0-034d-40f6-92f7-ff11eda0c760",
"tag": "HM2D9XElG3U=",
"name": "Low Cost Reflections",
"cpuTier": "BACKGROUND",
"maxAllowedRunningJobs": 10,
"maxStartTimeoutMs": 86400000
},
{
"id": "450ea2a5-9a64-4679-99cb-7b01bf6bba27",
"tag": "p22KaFcaB7g=",
"name": "COPY & OPTIMIZATION Queue",
"maxMemoryPerNodeBytes": 4294967296,
"maxQueryMemoryPerNodeBytes": 4294967296,
"cpuTier": "MEDIUM",
"maxAllowedRunningJobs": 2,
"maxStartTimeoutMs": 300000,
"engineId": "YARN"
},
{
"id": "a254d63e-9b0e-41be-af4a-1acc5bfe2332",
"tag": "//gNL3Ta2bY=",
"name": "Low Cost User Queries",
"cpuTier": "MEDIUM",
"maxAllowedRunningJobs": 100,
"maxStartTimeoutMs": 300000
},
{
"id": "c2917cce-b566-4c6a-be63-2e28488a6928",
"tag": "wa+vYmA73gU=",
"name": "High Cost User Queries",
"cpuTier": "MEDIUM",
"maxAllowedRunningJobs": 10,
"maxStartTimeoutMs": 300000
}
]
}
Queue Attributes
[Object]
List of queue objects in the Dremio instance.
data
id
String (UUID)
Unique identifier of the queue, in UTC format.
Example
1990e713-3cd2-458c-89e1-68995c2c1047
tag
String
Unique identifier of the version of the queue. Dremio changes the tag whenever the queue changes and uses the tag to ensure that PUT requests apply to the most recent version of the queue.
Example
BNGRmgfEnDg=
name
String
User-provided name for the queue.
Example
High Cost Reflections
maxMemoryPerNodeBytes
Integer
Total memory (in bytes) that all queries running in parallel in a given queue can use per executor node.
Example 8589934592
maxQueryMemoryPerNodeBytes
Integer
Total memory (in bytes) that each query in a given queue can use per executor node.
Example 8589934592
cpuTier
String
Amount of CPU time that threads get compared to other threads.
Enum
BACKGROUND
,
LOW
,
MEDIUM
,
HIGH
,
CRITICAL
Example
BACKGROUND
maxAllowedRunningJobs
Integer
Number of queries that are allowed to run in parallel.
Example 10
maxStartTimeoutMs
Integer
Maximum length of time that a query can wait in the queue before it is cancelled, in milliseconds.
Example 300000
maxRunTimeoutMs
Integer
Maximum length of time that a query can run before it is cancelled, in milliseconds.
Example 300000
engineId
String
Name of the execution engine to which the queue's queries are routed. If you do not specify an engineId, the queue's queries run on any engine that is available at the time of execution. The engineID attribute is omitted from the queue object if no engine is specified. For more information, read Workload Management-based Routing.
Example
DATA
Creating a Queue
Create a WLM queue.
Method and URLPOST /api/v3/wlm/queue
Parameters
name
String
User-provided name for the queue.
Example
High Cost Reflections
maxMemoryPerNodeBytes
Integer
Optional
Total memory (in bytes) that all queries running in parallel in a given queue can use per executor node.
Example 8589934592
maxQueryMemoryPerNodeBytes
Integer
Optional
Total memory (in bytes) that each query in a given queue can use per executor node.
Example 8589934592
cpuTier
String
Optional
Amount of CPU time that threads should get compared to other threads. Default is MEDIUM.
Enum
BACKGROUND
,
LOW
,
MEDIUM
,
HIGH
,
CRITICAL
Example
BACKGROUND
maxAllowedRunningJobs
Integer
Optional
Number of queries that are allowed to run in parallel.
Example 10
maxStartTimeoutMs
Integer
Optional
Maximum length of time that a query can wait in the queue before it is cancelled, in milliseconds.
Example 300000
maxRunTimeoutMs
Integer
Optional
Maximum length of time that a query can run before it is cancelled, in milliseconds.
Example 300000
engineId
String
Name of the execution engine to which the queue's queries should be routed. If you do not specify an engineId, the queue's queries run on any engine that is available at the time of execution. For more information, read Workload Management-based Routing.
Example
DATA
Example Request
curl -X POST 'https://{DREMIO_ORIGIN}/api/v3/wlm/queue' \
--header 'Authorization: _dremio{tokenstring}' \
--header 'Content-Type: application/json' \
--data-raw '{
"name": "High Cost Reflections",
"maxQueryMemoryPerNodeBytes": 8589934592,
"maxMemoryPerNodeBytes": 8589934592,
"cpuTier": "BACKGROUND",
"maxAllowedRunningJobs": 10,
"maxStartTimeoutMs": 300000,
"maxRunTimeoutMs": 300000,
"engineId": "DATA"
}'
{
"id": "1990e713-3cd2-458c-89e1-68995c2c1047",
"tag": "BNGRmgfEnDg=",
"name": "High Cost Reflections",
"maxMemoryPerNodeBytes": 8589934592,
"maxQueryMemoryPerNodeBytes": 8589934592,
"cpuTier": "BACKGROUND",
"maxAllowedRunningJobs": 10,
"maxStartTimeoutMs": 300000,
"maxRunTimeoutMs": 300000,
"engineId": "DATA"
}
Response Status Codes
200
OK
400
Invalid body
401
Unauthorized
403
User does not have permission to create this queue
404
Not Found
405
Method Not Allowed
409
Conflict, queue with specified name already exists
Retrieving All Queues
Retrieve all WLM queues.
Method and URLGET /api/v3/wlm/queue
curl -X GET 'https://{DREMIO_ORIGIN}/api/v3/wlm/queue' \
--header 'Authorization: _dremio{tokenstring}' \
--header 'Content-Type: application/json'
{
"data": [
{
"id": "1990e713-3cd2-458c-89e1-68995c2c1047",
"tag": "BNGRmgfEnDg=",
"name": "High Cost Reflections",
"maxMemoryPerNodeBytes": 8589934592,
"maxQueryMemoryPerNodeBytes": 8589934592,
"cpuTier": "BACKGROUND",
"maxAllowedRunningJobs": 10,
"maxStartTimeoutMs": 300000,
"maxRunTimeoutMs": 300000,
"engineId": "DATA"
},
{
"id": "0dbc50a0-034d-40f6-92f7-ff11eda0c760",
"tag": "HM2D9XElG3U=",
"name": "Low Cost Reflections",
"cpuTier": "BACKGROUND",
"maxAllowedRunningJobs": 10,
"maxStartTimeoutMs": 86400000
},
{
"id": "450ea2a5-9a64-4679-99cb-7b01bf6bba27",
"tag": "p22KaFcaB7g=",
"name": "COPY & OPTIMIZATION Queue",
"maxMemoryPerNodeBytes": 4294967296,
"maxQueryMemoryPerNodeBytes": 4294967296,
"cpuTier": "MEDIUM",
"maxAllowedRunningJobs": 2,
"maxStartTimeoutMs": 300000,
"engineId": "YARN"
},
{
"id": "a254d63e-9b0e-41be-af4a-1acc5bfe2332",
"tag": "//gNL3Ta2bY=",
"name": "Low Cost User Queries",
"cpuTier": "MEDIUM",
"maxAllowedRunningJobs": 100,
"maxStartTimeoutMs": 300000
},
{
"id": "c2917cce-b566-4c6a-be63-2e28488a6928",
"tag": "wa+vYmA73gU=",
"name": "High Cost User Queries",
"cpuTier": "MEDIUM",
"maxAllowedRunningJobs": 10,
"maxStartTimeoutMs": 300000
}
]
}
Response Status Codes
200
OK
401
Unauthorized
403
User does not have permission to access queues
404
Not Found
500
Internal Server Error
Retrieving a Queue by ID
Retrieve a specific WLM queue by the queue’s ID.
Method and URLGET /api/v3/wlm/queue/{id}
Parameters
id
path
String (UUID)
Unique identifier of the queue you want to retrieve, in UTC format.
Example
1990e713-3cd2-458c-89e1-68995c2c1047
Example Request
curl -X GET 'https://{DREMIO_ORIGIN}/api/v3/wlm/queue/1990e713-3cd2-458c-89e1-68995c2c1047' \
--header 'Authorization: _dremio{tokenstring}' \
--header 'Content-Type: application/json'
{
"id": "1990e713-3cd2-458c-89e1-68995c2c1047",
"tag": "BNGRmgfEnDg=",
"name": "High Cost Reflections",
"maxMemoryPerNodeBytes": 8589934592,
"maxQueryMemoryPerNodeBytes": 8589934592,
"cpuTier": "BACKGROUND",
"maxAllowedRunningJobs": 10,
"maxStartTimeoutMs": 300000,
"maxRunTimeoutMs": 300000,
"engineId": "DATA"
}
Response Status Codes
200
OK
401
Unauthorized
403
User does not have permission to access this queue
404
Queue does not exist
500
Internal Server Error
Retrieving a Queue by Name
Retrieve a specific WLM queue by the queue’s name.
Method and URLGET /api/v3/wlm/queue/by-name/{name}
Parameters
name
path
String
Name for the queue you want to retrieve. URI-encode the queue name to replace special characters with their UTF-8 characters, such as %3A for a colon and %20 for a space.
Example
High%20Cost%20Reflections
Example Request
curl -X GET 'https://{DREMIO_ORIGIN}/api/v3/wlm/queue/by-name/High%20Cost%20Reflections' \
--header 'Authorization: _dremio{tokenstring}' \
--header 'Content-Type: application/json'
{
"id": "1990e713-3cd2-458c-89e1-68995c2c1047",
"tag": "BNGRmgfEnDg=",
"name": "High Cost Reflections",
"maxMemoryPerNodeBytes": 8589934592,
"maxQueryMemoryPerNodeBytes": 8589934592,
"cpuTier": "BACKGROUND",
"maxAllowedRunningJobs": 10,
"maxStartTimeoutMs": 300000,
"maxRunTimeoutMs": 300000,
"engineId": "DATA"
}
Response Status Codes
200
OK
401
Unauthorized
403
User does not have permission to access this queue
404
Queue does not exist
500
Internal Server Error
Updating a Queue
Update the specified WLM queue.
Method and URLPUT /api/v3/wlm/queue/{id}
Parameters
id
path
String (UUID)
Unique identifier of the queue you want to update, in UTC format.
Example
1990e713-3cd2-458c-89e1-68995c2c1047
tag
body
String
Unique identifier of the version of the queue to update. Dremio uses the tag to ensure that you are updating the most recent version of the queue.
Example
BNGRmgfEnDg=
name
String
User-provided name for the queue.
Example
High Cost Reflections
maxMemoryPerNodeBytes
body
Integer
Optional
Total memory (in bytes) that all queries running in parallel in a given queue can use per executor node. If you omit the maxMemoryPerNodeBytes parameter in a PUT request, Dremio removes the existing maxMemoryPerNodeBytes value from the queue. To keep the existing value while making other updates, include the existing maxMemoryPerNodeBytes parameter and value in the PUT request.
Example 8589934592
maxQueryMemoryPerNodeBytes
body
Integer
Optional
Total memory (in bytes) that each query in a given queue can use per executor node. If you omit the maxQueryMemoryPerNodeBytes parameter in a PUT request, Dremio removes the existing maxQueryMemoryPerNodeBytes value from the queue. To keep the existing value while making other updates, include the existing maxQueryMemoryPerNodeBytes parameter and value in the PUT request.
Example 8589934592
cpuTier
body
String
Optional
Amount of CPU time that threads should get compared to other threads. Default is MEDIUM. If you omit the cpuTier parameter in a PUT request, Dremio replaces it with the default value. To keep the existing setting while making other updates, include the existing cpuTier parameter and setting in the PUT request.
Enum
BACKGROUND
,
LOW
,
MEDIUM
,
HIGH
,
CRITICAL
Example
LOW
maxAllowedRunningJobs
body
Integer
Optional
Number of queries that are allowed to run in parallel. If you omit the maxAllowedRunningJobs parameter in a PUT request, Dremio removes the existing maxAllowedRunningJobs value from the queue. To keep the existing value while making other updates, include the existing maxAllowedRunningJobs parameter and value in the PUT request.
Example 100
maxStartTimeoutMs
body
Integer
Optional
Maximum length of time that a query can wait in the queue before it is cancelled, in milliseconds. If you omit the maxStartTimeoutMs parameter in a PUT request, Dremio removes the existing maxStartTimeoutMs setting from the queue. To keep the existing setting while making other updates, include the existing maxStartTimeoutMs parameter and setting in the PUT request.
Example 300000
maxRunTimeoutMs
body
Integer
Optional
Maximum length of time that a query is allowed to run before it is cancelled, in milliseconds. If you omit the maxRunTimeoutMs parameter in a PUT request, Dremio removes the existing maxRunTimeoutMs value from the queue. To keep the existing value while making other updates, include the existing maxRunTimeoutMs parameter and value in the PUT request.
Example 300000
engineId
String
Name of the execution engine to which the queue's queries should be routed. If you do not specify an engineId, the queue's queries run on any engine that is available at the time of execution. For more information, read Workload Management-based Routing.
Example
DATA
Example Request
curl -X PUT 'https://{DREMIO_ORIGIN}/api/v3/wlm/queue/1990e713-3cd2-458c-89e1-68995c2c1047' \
--header 'Authorization: _dremio{tokenstring}' \
--header 'Content-Type: application/json' \
--data-raw '{
"tag": "BNGRmgfEnDg=",
"name": "High Cost Reflections",
"maxMemoryPerNodeBytes": 8589934592,
"maxQueryMemoryPerNodeBytes": 8589934592,
"cpuTier": "LOW",
"maxAllowedRunningJobs": 100,
"maxStartTimeoutMs": 300000,
"maxRunTimeoutMs": 300000,
"engineId": "DATA"
}'
{
"id": "1990e713-3cd2-458c-89e1-68995c2c1047",
"tag": "xQh6KNyEjus=",
"name": "High Cost Reflections",
"maxMemoryPerNodeBytes": 8589934592,
"maxQueryMemoryPerNodeBytes": 8589934592,
"cpuTier": "LOW",
"maxAllowedRunningJobs": 100,
"maxStartTimeoutMs": 300000,
"maxRunTimeoutMs": 300000,
"engineId": "DATA"
}
Response Status Codes
200
OK
400
Invalid body
401
Unauthorized
403
User does not have permission to update this queue
404
Queue doesn't exist
409
Conflict, queue has been modified (version mismatch)
500
Internal Server Error
Deleting a Queue
Delete the specified WLM queue.
Method and URLDELETE /api/v3/wlm/queue/{id}
Parameters
id
path
String (UUID)
Unique identifier of the queue that you want to delete.
Example
1990e713-3cd2-458c-89e1-68995c2c1047
Example Request
curl -X DELETE 'https://{DREMIO_ORIGIN}/api/v3/wlm/queue/1990e713-3cd2-458c-89e1-68995c2c1047' \
--header 'Authorization: _dremio{tokenstring}' \
--header 'Content-Type: application/json'
No response
Response Status Codes
200
OK
400
Queue still has rules associated with it; queue is not empty
401
Unauthorized
403
User does not have permission to access this queue
404
Queue doesn’t exist
405
Method Not Allowed