Community Edition only: For the Enterprise Edition equivalent, see Workload Management.
Dremio has 3 static job classes for CPU slicing. Here they are in descending priority order:
- Near-realtime - This class includes all UI preview jobs. This is to ensure a responsive user experience for all operations in Dremio’s UI.
- Interactive - This class is for all user jobs including ones submitted over ODBC and JDBC as well as full runs in the UI.
- Background - This class includes all Reflection creation and maintenance jobs. These jobs have the lowest priority to ensure higher performance for user-submitted jobs.
Query Queue Control
Dremio supports placing user jobs into two static job queues (Large and Small) based on a configurable query cost threshold.
Maximum concurrent queries and queue timeout can be set
for each queue under
Admin > Cluster > Queue Control > Query Queue Control.
We recommend configuring concurrency of the large query queue to be lower than that of the small query queue.
This ensures that at any given time, only a few large jobs are running and allow for multiple smaller queries to complete.
Queue timeout is configurable with a default of five (5) minutes.
Reflection Queue Control
Dremio supports placing reflection jobs into two static job queues (Large and Small)
Admin > Cluster > Queue Control > Reflection Queue Control.
This ensures that at any given time, only a minimum if large reflection jobs are running and allow for multiple smaller queries to complete.
Reflection queue timeout is configurable with a default of one (1) day.
Query Memory Control
Maximum memory per query can also be configured per job type
Admin > Cluster > Queue Control > Query Memory Control:
The query threshold is established by analyzing the overall cost for a variety of workloads and setting the threshold based on expected workloads, SLAs, and available cluster resources. The default threshold is 30,000,000 and is a large query threshold or plan cost.
The query threshold can be configured under
Admin > Cluster > Queue Control > Query Thresholds:
The query cost for a specific job can be found under
Jobs > Profile > Resource Allocation > Query Cost:
The overall query cost used for placing jobs in small vs. large queues is computed
by summing up
rowcount value for all operators (each line) in the physical plan for a given job.