Job Queues

Job Classes

Dremio has 3 static job classes for CPU slicing. Here they are in descending priority order:

  • Near-realtime - This class includes all UI preview jobs. This is to ensure a responsive user experience for all operations in Dremio's UI.
  • Interactive - This class if for all user jobs including ones submitted over ODBC and JDBC as well as full runs in the UI.
  • Background - This class includes all Reflection creation and maintenance jobs. These jobs have the lowest priority to ensure higher performance for user-submitted jobs.

Static Job Queues

Dremio supports placing user jobs into two static job queues (Large and Small) based on a configurable query cost threshold.

Maximum concurrent queries and queue timeout can be set for each queue under Admin > Advanced Settings > Query Queue Control. We recommend configuring concurrency of the large query queue to be lower than that of the small query queue. This will ensure that at any given time, only a few large jobs are running and allow for multiple smaller queries to complete.

Maximum memory per query can also be configured per job type under Admin > Advanced Settings > Query Memory Control:

Determining Query Cost Threshold

Overall query cost used for placing jobs in small vs. large queues is computed by summing up rowcount value for all operators (each line) in the physical plan for a given job. This can be found under Jobs > Planning > Final Physical Transformation:

We recommend calculate and analyze the overall cost for a variety of workloads and setting the exact threshold based on expected workloads, SLAs and available cluster resources.

Based on this analysis, the threshold can be configured under Admin > Advanced Settings > Query Thresholds:

results matching ""

    No results matching ""