Dremio Cloud
Dremio Software
 
On this page

    Queue Control

    note:

    Community Edition only: For the Enterprise Edition equivalent, see Workload Management.

    Job Classes

    Dremio has 3 static job classes for CPU slicing. Here they are in descending priority order:

    • Near-realtime - This class includes all UI preview jobs. This is to ensure a responsive user experience for all operations in Dremio’s UI.
    • Interactive - This class is for all user jobs including ones submitted over ODBC and JDBC as well as full runs in the UI.
    • Background - This class includes all Reflection creation and maintenance jobs. These jobs have the lowest priority to ensure higher performance for user-submitted jobs.

    Query Queue Control

    Dremio supports placing user jobs into two static job queues (Large and Small) based on a configurable query cost threshold.

    Maximum concurrent queries and queue timeout can be set for each queue under Admin > Cluster > Queue Control > Query Queue Control. We recommend configuring concurrency of the large query queue to be lower than that of the small query queue. This ensures that at any given time, only a few large jobs are running and allow for multiple smaller queries to complete.

    Queue timeout is configurable with a default of five (5) minutes.

    Reflection Queue Control

    Dremio supports placing reflection jobs into two static job queues (Large and Small) under Admin > Cluster > Queue Control > Reflection Queue Control.

    This ensures that at any given time, only a minimum if large reflection jobs are running and allow for multiple smaller queries to complete.

    Reflection queue timeout is configurable with a default of one (1) day.

    Query Memory Control

    Maximum memory per query can also be configured per job type under Admin > Cluster > Queue Control > Query Memory Control:

    Query Threshold

    The query threshold is established by analyzing the overall cost for a variety of workloads and setting the threshold based on expected workloads, SLAs, and available cluster resources. The default threshold is 30,000,000 and is a large query threshold or plan cost.

    The query threshold can be configured under Admin > Cluster > Queue Control > Query Thresholds:

    The query cost for a specific job can be found under Jobs > Profile > Resource Allocation > Query Cost:

    note:

    The overall query cost used for placing jobs in small vs. large queues is computed by summing up rowcount value for all operators (each line) in the physical plan for a given job.