On this page

    Queue enterprise

    Use the Workload Management (WLM) API to create, retrieve, update, and delete WLM queues.

    Queue Object
    {
      "data": [
        {
          "id": "1990e713-3cd2-458c-89e1-68995c2c1047",
          "tag": "BNGRmgfEnDg=",
          "name": "High Cost Reflections",
          "maxMemoryPerNodeBytes": 8589934592,
          "maxQueryMemoryPerNodeBytes": 8589934592,
          "cpuTier": "BACKGROUND",
          "maxAllowedRunningJobs": 10,
          "maxStartTimeoutMs": 300000,
          "maxRunTimeoutMs": 300000,
          "engineId": "DATA"
        },
        {
          "id": "0dbc50a0-034d-40f6-92f7-ff11eda0c760",
          "tag": "HM2D9XElG3U=",
          "name": "Low Cost Reflections",
          "cpuTier": "BACKGROUND",
          "maxAllowedRunningJobs": 10,
          "maxStartTimeoutMs": 86400000
        },
        {
          "id": "450ea2a5-9a64-4679-99cb-7b01bf6bba27",
          "tag": "p22KaFcaB7g=",
          "name": "COPY & OPTIMIZATION Queue",
          "maxMemoryPerNodeBytes": 4294967296,
          "maxQueryMemoryPerNodeBytes": 4294967296,
          "cpuTier": "MEDIUM",
          "maxAllowedRunningJobs": 2,
          "maxStartTimeoutMs": 300000,
          "engineId": "YARN"
        },
        {
          "id": "a254d63e-9b0e-41be-af4a-1acc5bfe2332",
          "tag": "//gNL3Ta2bY=",
          "name": "Low Cost User Queries",
          "cpuTier": "MEDIUM",
          "maxAllowedRunningJobs": 100,
          "maxStartTimeoutMs": 300000
        },
        {
          "id": "c2917cce-b566-4c6a-be63-2e28488a6928",
          "tag": "wa+vYmA73gU=",
          "name": "High Cost User Queries",
          "cpuTier": "MEDIUM",
          "maxAllowedRunningJobs": 10,
          "maxStartTimeoutMs": 300000
        }
      ]
    }
    

    Queue Attributes

    data

    [Object]

    List of queue objects in the Dremio instance.

    data

    id

    String (UUID)

    Unique identifier of the queue, in UTC format.

    Example 1990e713-3cd2-458c-89e1-68995c2c1047


    tag

    String

    Unique identifier of the version of the queue. Dremio changes the tag whenever the queue changes and uses the tag to ensure that PUT requests apply to the most recent version of the queue.

    Example BNGRmgfEnDg=


    name

    String

    User-provided name for the queue.

    Example High Cost Reflections


    maxMemoryPerNodeBytes

    Integer

    Total memory (in bytes) that all queries running in parallel in a given queue can use per executor node.

    Example 8589934592


    maxQueryMemoryPerNodeBytes

    Integer

    Total memory (in bytes) that each query in a given queue can use per executor node.

    Example 8589934592


    cpuTier

    String

    Amount of CPU time that threads get compared to other threads.

    Enum BACKGROUND , LOW , MEDIUM , HIGH , CRITICAL

    Example BACKGROUND


    maxAllowedRunningJobs

    Integer

    Number of queries that are allowed to run in parallel.

    Example 10


    maxStartTimeoutMs

    Integer

    Maximum length of time that a query can wait in the queue before it is cancelled, in milliseconds.

    Example 300000


    maxRunTimeoutMs

    Integer

    Maximum length of time that a query can run before it is cancelled, in milliseconds.

    Example 300000


    engineId

    String

    Name of the execution engine to which the queue's queries are routed. If you do not specify an engineId, the queue's queries run on any engine that is available at the time of execution. The engineID attribute is omitted from the queue object if no engine is specified. For more information, read Workload Management-based Routing.

    Example DATA

    Creating a Queue

    Create a WLM queue.

    Method and URL
    POST /api/v3/wlm/queue
    

    Parameters

    name

    String

    User-provided name for the queue.

    Example High Cost Reflections


    maxMemoryPerNodeBytes

    Integer

    Optional

    Total memory (in bytes) that all queries running in parallel in a given queue can use per executor node.

    Example 8589934592


    maxQueryMemoryPerNodeBytes

    Integer

    Optional

    Total memory (in bytes) that each query in a given queue can use per executor node.

    Example 8589934592


    cpuTier

    String

    Optional

    Amount of CPU time that threads should get compared to other threads. Default is MEDIUM.

    Enum BACKGROUND , LOW , MEDIUM , HIGH , CRITICAL

    Example BACKGROUND


    maxAllowedRunningJobs

    Integer

    Optional

    Number of queries that are allowed to run in parallel.

    Example 10


    maxStartTimeoutMs

    Integer

    Optional

    Maximum length of time that a query can wait in the queue before it is cancelled, in milliseconds.

    Example 300000


    maxRunTimeoutMs

    Integer

    Optional

    Maximum length of time that a query can run before it is cancelled, in milliseconds.

    Example 300000


    engineId

    String

    Name of the execution engine to which the queue's queries should be routed. If you do not specify an engineId, the queue's queries run on any engine that is available at the time of execution. For more information, read Workload Management-based Routing.

    Example DATA


    Example Request
    curl -X POST 'https://{DREMIO_ORIGIN}/api/v3/wlm/queue' \
    --header 'Authorization: _dremio{tokenstring}' \
    --header 'Content-Type: application/json' \
    --data-raw '{
      "name": "High Cost Reflections",
      "maxQueryMemoryPerNodeBytes": 8589934592,
      "maxMemoryPerNodeBytes": 8589934592,
      "cpuTier": "BACKGROUND",
      "maxAllowedRunningJobs": 10,
      "maxStartTimeoutMs": 300000,
      "maxRunTimeoutMs": 300000,
      "engineId": "DATA"
    }'
    
    Example Response
    {
      "id": "1990e713-3cd2-458c-89e1-68995c2c1047",
      "tag": "BNGRmgfEnDg=",
      "name": "High Cost Reflections",
      "maxMemoryPerNodeBytes": 8589934592,
      "maxQueryMemoryPerNodeBytes": 8589934592,
      "cpuTier": "BACKGROUND",
      "maxAllowedRunningJobs": 10,
      "maxStartTimeoutMs": 300000,
      "maxRunTimeoutMs": 300000,
      "engineId": "DATA"
    }
    

    Response Status Codes

    200

    OK

    400

    Invalid body

    401

    Unauthorized

    403

    User does not have permission to create this queue

    404

    Not Found

    405

    Method Not Allowed

    409

    Conflict, queue with specified name already exists


    Retrieving All Queues

    Retrieve all WLM queues.

    Method and URL
    GET /api/v3/wlm/queue
    
    Example Request
    curl -X GET 'https://{DREMIO_ORIGIN}/api/v3/wlm/queue' \
    --header 'Authorization: _dremio{tokenstring}' \
    --header 'Content-Type: application/json'
    
    Example Response
    {
      "data": [
        {
          "id": "1990e713-3cd2-458c-89e1-68995c2c1047",
          "tag": "BNGRmgfEnDg=",
          "name": "High Cost Reflections",
          "maxMemoryPerNodeBytes": 8589934592,
          "maxQueryMemoryPerNodeBytes": 8589934592,
          "cpuTier": "BACKGROUND",
          "maxAllowedRunningJobs": 10,
          "maxStartTimeoutMs": 300000,
          "maxRunTimeoutMs": 300000,
          "engineId": "DATA"
        },
        {
          "id": "0dbc50a0-034d-40f6-92f7-ff11eda0c760",
          "tag": "HM2D9XElG3U=",
          "name": "Low Cost Reflections",
          "cpuTier": "BACKGROUND",
          "maxAllowedRunningJobs": 10,
          "maxStartTimeoutMs": 86400000
        },
        {
          "id": "450ea2a5-9a64-4679-99cb-7b01bf6bba27",
          "tag": "p22KaFcaB7g=",
          "name": "COPY & OPTIMIZATION Queue",
          "maxMemoryPerNodeBytes": 4294967296,
          "maxQueryMemoryPerNodeBytes": 4294967296,
          "cpuTier": "MEDIUM",
          "maxAllowedRunningJobs": 2,
          "maxStartTimeoutMs": 300000,
          "engineId": "YARN"
        },
        {
          "id": "a254d63e-9b0e-41be-af4a-1acc5bfe2332",
          "tag": "//gNL3Ta2bY=",
          "name": "Low Cost User Queries",
          "cpuTier": "MEDIUM",
          "maxAllowedRunningJobs": 100,
          "maxStartTimeoutMs": 300000
        },
        {
          "id": "c2917cce-b566-4c6a-be63-2e28488a6928",
          "tag": "wa+vYmA73gU=",
          "name": "High Cost User Queries",
          "cpuTier": "MEDIUM",
          "maxAllowedRunningJobs": 10,
          "maxStartTimeoutMs": 300000
        }
      ]
    }
    

    Response Status Codes

    200

    OK

    401

    Unauthorized

    403

    User does not have permission to access queues

    404

    Not Found

    500

    Internal Server Error


    Retrieving a Queue by ID

    Retrieve a specific WLM queue by the queue’s ID.

    Method and URL
    GET /api/v3/wlm/queue/{id}
    

    Parameters

    id

    path

    String (UUID)

    Unique identifier of the queue you want to retrieve, in UTC format.

    Example 1990e713-3cd2-458c-89e1-68995c2c1047


    Example Request
    curl -X GET 'https://{DREMIO_ORIGIN}/api/v3/wlm/queue/1990e713-3cd2-458c-89e1-68995c2c1047' \
    --header 'Authorization: _dremio{tokenstring}' \
    --header 'Content-Type: application/json'
    
    Example Response
    {
      "id": "1990e713-3cd2-458c-89e1-68995c2c1047",
      "tag": "BNGRmgfEnDg=",
      "name": "High Cost Reflections",
      "maxMemoryPerNodeBytes": 8589934592,
      "maxQueryMemoryPerNodeBytes": 8589934592,
      "cpuTier": "BACKGROUND",
      "maxAllowedRunningJobs": 10,
      "maxStartTimeoutMs": 300000,
      "maxRunTimeoutMs": 300000,
      "engineId": "DATA"
    }
    

    Response Status Codes

    200

    OK

    401

    Unauthorized

    403

    User does not have permission to access this queue

    404

    Queue does not exist

    500

    Internal Server Error


    Retrieving a Queue by Name

    Retrieve a specific WLM queue by the queue’s name.

    Method and URL
    GET /api/v3/wlm/queue/by-name/{name}
    

    Parameters

    name

    path

    String

    Name for the queue you want to retrieve. URI-encode the queue name to replace special characters with their UTF-8 characters, such as %3A for a colon and %20 for a space.

    Example High%20Cost%20Reflections


    Example Request
    curl -X GET 'https://{DREMIO_ORIGIN}/api/v3/wlm/queue/by-name/High%20Cost%20Reflections' \
    --header 'Authorization: _dremio{tokenstring}' \
    --header 'Content-Type: application/json'
    
    Example Response
    {
      "id": "1990e713-3cd2-458c-89e1-68995c2c1047",
      "tag": "BNGRmgfEnDg=",
      "name": "High Cost Reflections",
      "maxMemoryPerNodeBytes": 8589934592,
      "maxQueryMemoryPerNodeBytes": 8589934592,
      "cpuTier": "BACKGROUND",
      "maxAllowedRunningJobs": 10,
      "maxStartTimeoutMs": 300000,
      "maxRunTimeoutMs": 300000,
      "engineId": "DATA"
    }
    

    Response Status Codes

    200

    OK

    401

    Unauthorized

    403

    User does not have permission to access this queue

    404

    Queue does not exist

    500

    Internal Server Error


    Updating a Queue

    Update the specified WLM queue.

    Method and URL
    PUT /api/v3/wlm/queue/{id}
    

    Parameters

    id

    path

    String (UUID)

    Unique identifier of the queue you want to update, in UTC format.

    Example 1990e713-3cd2-458c-89e1-68995c2c1047


    tag

    body

    String

    Unique identifier of the version of the queue to update. Dremio uses the tag to ensure that you are updating the most recent version of the queue.

    Example BNGRmgfEnDg=


    name

    String

    User-provided name for the queue.

    Example High Cost Reflections


    maxMemoryPerNodeBytes

    body

    Integer

    Optional

    Total memory (in bytes) that all queries running in parallel in a given queue can use per executor node. If you omit the maxMemoryPerNodeBytes parameter in a PUT request, Dremio removes the existing maxMemoryPerNodeBytes value from the queue. To keep the existing value while making other updates, include the existing maxMemoryPerNodeBytes parameter and value in the PUT request.

    Example 8589934592


    maxQueryMemoryPerNodeBytes

    body

    Integer

    Optional

    Total memory (in bytes) that each query in a given queue can use per executor node. If you omit the maxQueryMemoryPerNodeBytes parameter in a PUT request, Dremio removes the existing maxQueryMemoryPerNodeBytes value from the queue. To keep the existing value while making other updates, include the existing maxQueryMemoryPerNodeBytes parameter and value in the PUT request.

    Example 8589934592


    cpuTier

    body

    String

    Optional

    Amount of CPU time that threads should get compared to other threads. Default is MEDIUM. If you omit the cpuTier parameter in a PUT request, Dremio replaces it with the default value. To keep the existing setting while making other updates, include the existing cpuTier parameter and setting in the PUT request.

    Enum BACKGROUND , LOW , MEDIUM , HIGH , CRITICAL

    Example LOW


    maxAllowedRunningJobs

    body

    Integer

    Optional

    Number of queries that are allowed to run in parallel. If you omit the maxAllowedRunningJobs parameter in a PUT request, Dremio removes the existing maxAllowedRunningJobs value from the queue. To keep the existing value while making other updates, include the existing maxAllowedRunningJobs parameter and value in the PUT request.

    Example 100


    maxStartTimeoutMs

    body

    Integer

    Optional

    Maximum length of time that a query can wait in the queue before it is cancelled, in milliseconds. If you omit the maxStartTimeoutMs parameter in a PUT request, Dremio removes the existing maxStartTimeoutMs setting from the queue. To keep the existing setting while making other updates, include the existing maxStartTimeoutMs parameter and setting in the PUT request.

    Example 300000


    maxRunTimeoutMs

    body

    Integer

    Optional

    Maximum length of time that a query is allowed to run before it is cancelled, in milliseconds. If you omit the maxRunTimeoutMs parameter in a PUT request, Dremio removes the existing maxRunTimeoutMs value from the queue. To keep the existing value while making other updates, include the existing maxRunTimeoutMs parameter and value in the PUT request.

    Example 300000


    engineId

    String

    Name of the execution engine to which the queue's queries should be routed. If you do not specify an engineId, the queue's queries run on any engine that is available at the time of execution. For more information, read Workload Management-based Routing.

    Example DATA


    Example Request
    curl -X PUT 'https://{DREMIO_ORIGIN}/api/v3/wlm/queue/1990e713-3cd2-458c-89e1-68995c2c1047' \
    --header 'Authorization: _dremio{tokenstring}' \
    --header 'Content-Type: application/json' \
    --data-raw '{
      "tag": "BNGRmgfEnDg=",
      "name": "High Cost Reflections",
      "maxMemoryPerNodeBytes": 8589934592,
      "maxQueryMemoryPerNodeBytes": 8589934592,
      "cpuTier": "LOW",
      "maxAllowedRunningJobs": 100,
      "maxStartTimeoutMs": 300000,
      "maxRunTimeoutMs": 300000,
      "engineId": "DATA"
    }'
    
    Example Response
    {
      "id": "1990e713-3cd2-458c-89e1-68995c2c1047",
      "tag": "xQh6KNyEjus=",
      "name": "High Cost Reflections",
      "maxMemoryPerNodeBytes": 8589934592,
      "maxQueryMemoryPerNodeBytes": 8589934592,
      "cpuTier": "LOW",
      "maxAllowedRunningJobs": 100,
      "maxStartTimeoutMs": 300000,
      "maxRunTimeoutMs": 300000,
      "engineId": "DATA"
    }
    

    Response Status Codes

    200

    OK

    400

    Invalid body

    401

    Unauthorized

    403

    User does not have permission to update this queue

    404

    Queue doesn't exist

    409

    Conflict, queue has been modified (version mismatch)

    500

    Internal Server Error


    Deleting a Queue

    Delete the specified WLM queue.

    Method and URL
    DELETE /api/v3/wlm/queue/{id}
    

    Parameters

    id

    path

    String (UUID)

    Unique identifier of the queue that you want to delete.

    Example 1990e713-3cd2-458c-89e1-68995c2c1047


    Example Request
    curl -X DELETE 'https://{DREMIO_ORIGIN}/api/v3/wlm/queue/1990e713-3cd2-458c-89e1-68995c2c1047' \
    --header 'Authorization: _dremio{tokenstring}' \
    --header 'Content-Type: application/json'
    
    Example Response
    No response
    

    Response Status Codes

    200

    OK

    400

    Queue still has rules associated with it; queue is not empty

    401

    Unauthorized

    403

    User does not have permission to access this queue

    404

    Queue doesn’t exist

    405

    Method Not Allowed