You can use an SQL command to route queries directly to specified queues, so that you can isolate jobs from queues being used for other jobs or direct types of jobs to specific queues.
Routing Reflection-Refresh Jobs
Run this SQL command to route refresh jobs for all reflections on a dataset (either a table or a view) to a specific queue.
Datasets do not inherit this setting. For example, if for Table 1 you set the queue to Queue 1, refresh jobs for reflections on views that are created on Table 1 do not run in Queue 1 by default.
ALTER DATASET "<path>"."<dataset_name>" ROUTE REFLECTIONS TO <[DEFAULT QUEUE]|[QUEUE "[<queue_name> |<queue_uuid>]"]>
The context of the table or view. The context of a table is a location in a datasource or within a homespace in Dremio. The context of a view is a space within Dremio.
The name of the table or view.
The name of the queue to route jobs to.
The UUID of the queue to route jobs to.
Suppose that you had the following tables, views, and reflections, as shown in the first diagram:
- Table 1 has a reflection on it.
- Views 1 and 2 are derived from Table 1.
- View 1 has two reflections.
- Views 3 and 4 are derived from View 2.
- Views 3 and 4 have one reflection each.
You run these SQL commands to route the reflection-refresh jobs for Table 1, View 1, and View 3 to Queue 1. You let Dremio decide which of the two default queues for reflection-refresh jobs (Low-Cost Reflections and High-Cost Reflections) to use for View 4:
ALTER DATASET "Table 1" ROUTE REFLECTIONS TO QUEUE "Queue 1" ALTER DATASET "View 1" ROUTE REFLECTIONS TO QUEUE "Queue 1" ALTER DATASET "View 3" ROUTE REFLECTIONS TO QUEUE "Queue 1"
View 1 and View 3 do not inherit the setting from Table 1. If you want to run refresh jobs for the reflections on those views in Queue 1, you must run the SQL command on those views and specify Queue 1.
According to the schedule defined on the Reflection Refresh page of the settings for Table 1, the data in Table 1 is refreshed:
After the refresh of the data in Table 1 is complete, Dremio runs the reflection-refresh jobs for the reflections. The jobs for the reflections for Table 1, View 1, and View 3 run in Queue 1. The job for the reflection on View 4 runs in the default queue that Dremio selected: