Skip to main content
Version: 24.3.x

Queue Control

note

This feature is for Community Edition only. For Enterprise Edition, see Workload Management.

Job Classes​

Dremio has 3 static job classes for CPU slicing. Here they are in descending priority order:

  • Near-realtime - This class includes all UI preview jobs. This is to ensure a responsive user experience for all operations in Dremio's UI.
  • Interactive - This class is for all user jobs including ones submitted over ODBC and JDBC as well as full runs in the UI.
  • Background - This class includes all Reflection creation and maintenance jobs. These jobs have the lowest priority to ensure higher performance for user-submitted jobs.

Query Queue Control​

Dremio supports placing user jobs into two static job queues (Large and Small) based on a configurable query cost threshold.

Maximum concurrent queries and queue timeout can be set for each queue under Admin > Cluster > Queue Control > Query Queue Control. We recommend configuring concurrency of the large query queue to be lower than that of the small query queue. This ensures that at any given time, only a few large jobs are running and allow for multiple smaller queries to complete.

Queue timeout is configurable with a default of five (5) minutes.

Reflection Queue Control​

Dremio supports placing reflection jobs into two static job queues (Large and Small) under Admin > Cluster > Queue Control > Reflection Queue Control.

This ensures that at any given time, only a minimum if large reflection jobs are running and allow for multiple smaller queries to complete.

Reflection queue timeout is configurable with a default of one (1) day.

Query Memory Control​

Maximum memory per query can also be configured per job type under Admin > Cluster > Queue Control > Query Memory Control:

Query Threshold​

The query threshold is established by analyzing the overall cost for a variety of workloads and setting the threshold based on expected workloads, SLAs, and available cluster resources. The default threshold is 30,000,000 and is a large query threshold or plan cost.

The query threshold can be configured under Admin > Cluster > Queue Control > Query Thresholds:

The query cost for a specific job can be found under Jobs > Profile > Resource Allocation > Query Cost:

note

The overall query cost used for placing jobs in small vs. large queues is computed by summing up rowcount value for all operators (each line) in the physical plan for a given job.