Community Edition only
For the Enterprise Edition equivalent, see Workload Management.
Dremio has 3 static job classes for CPU slicing. Here they are in descending priority order:
Dremio supports placing user jobs into two static job queues (Large and Small) based on a configurable query cost threshold.
Maximum concurrent queries and queue timeout can be set
for each queue under Admin > Cluster > Queue Control > Query Queue Control
.
We recommend configuring concurrency of the large query queue to be lower than that of the small query queue.
This ensures that at any given time, only a few large jobs are running and allow for multiple smaller queries to complete.
Queue timeout is configurable with a default of five (5) minutes.
Dremio supports placing reflection jobs into two static job queues (Large and Small)
under Admin > Cluster > Queue Control > Reflection Queue Control
.
This ensures that at any given time, only a minimum if large reflection jobs are running and allow for multiple smaller queries to complete.
Reflection queue timeout is configurable with a default of one (1) day.
Maximum memory per query can also be configured per job type
under Admin > Cluster > Queue Control > Query Memory Control
:
The query threshold is established by analyzing the overall cost for a variety of workloads and setting the threshold based on expected workloads, SLAs, and available cluster resources. The default threshold is 30,000,000 and is a large query threshold or plan cost.
The query threshold can be configured under Admin > Cluster > Queue Control > Query Thresholds
:
The query cost for a specific job can be found under Jobs > Profile > Resource Allocation > Query Cost
:
The overall query cost used for placing jobs in small vs. large queues is computed by summing up
rowcount
value for all operators (each line) in the physical plan for a given job.