Viewing Jobs and Job Details
Dremio Cloud performs these types of task through jobs:
- Refreshing metadata
- Creating, revising, refreshing, and removing reflections
- Running queries submitted through the Dremio UI, JDBC, ODBC, or REST APIs
Job Attributes
Each job has the following attributes, which can appear in columns in the list of jobs:
- (Accelerated)
- The heading for this column is a purple lightning bolt. A lightning bolt in a row indicates that the query that the job ran was accelerated by one or more reflections.
- Dataset
- The dataset queried, if one was queried.
- Duration
- The length of time (in seconds) that a job required from start to completion.
- Job ID
- A universally unique identifier.
- Planner Cost Estimate
- A cost estimate calculated by Dremio based on an evaluation of the resources that to be used in the execution of a query. The number is not in units, and is intended to give a an idea of the cost of executing a query relative to the costs of executing other queries. Values are derived by adding weighted estimates of required I/O, memory, and CPU load. In reported values, K = thousand, M = million, B = billion, and T = trillion. For example, a value of 12,543,765,321 is reported as 12.5B.
- Planning Time
- The length of time (in seconds) in which the query optimizer planned the execution of the query.
- Query Type
- Represents the query type. For descriptions of the types, see Job Types.
- Start Time
- The date and time at which the job began.
- (Status)
- An icon that represents the current status of the job. A status represents one or more job states. For descriptions, see Job States and Statuses.
- User
- Username of the user who ran the query and initiated the job.
Job States and Statuses
Each job passes through a sequence of states until it is complete, though the sequence can be interrupted if a query is cancelled or if there is an error during a state. In this diagram, the states that a job passes through are in white, and the possible end states are in dark gray.
| State | Description |
|---|---|
| Pending | Represents a state where the query is waiting to be scheduled on the query pool. |
| Metadata Retrieval | Represents a state where metadata schema is retrieved and the SQL command is parsed. |
| Planning | Represents a state where the following are done:
|
| Engine Start | Represents a state where the engine starts if it has stopped. If the engine is stopped, it takes time to restart for the executors to be active. If the engine is already started, then this state does not have a duration. |
| Queued | Represents a state where a job is queued. Each queue has a limit of concurrent queries. If the queries in progress exceed the concurrency limit, the query should wait in the queue until the jobs in progress complete. |
| Execution Planning | Represents a state where executor nodes are selected from the chosen engine to run the query, and work is distributed to each executor. |
| Running | Represents a state where executor nodes execute and complete the fragments assigned to them. Typically, most queries spend more time in this state. |
| Canceled | Represents a terminal state that indicates that the query is canceled by the user or an intervention in the system. |
| Completed | Represents a terminal state that indicates that the query is successfully completed. |
| Failed | Represents a terminal state that indicates that the query has failed due to an error. |
This table lists the statuses that the UI lets you filter on and shows how they map to the states:
| Status | State |
|---|---|
| Setup | Pending |
| Metadata Retrieval | |
| Planning | |
| Engine Start | Engine Start |
| Queued | Queued |
| Running | Execution Planning |
| Starting | |
| Running | |
| Canceled | Canceled |
| Completed | Completed |
| Failed | Failed |
Job Types
There are five types of jobs:
| Type | Description |
|---|---|
| Accelerator | Jobs related to creating, maintaining, and removing reflections. |
| Downloads | Jobs that are used to download datasets. |
| External Tools | Jobs due to queries from client applications, such as Microsoft Power BI, Superset, Tableau, other third-party client applications, and custom applications. |
| Internal | Jobs for queries that Dremio Cloud submits for internal operations. |
| UI | Jobs due to queries issued from the SQL Runner in the Dremio Cloud UI. |
Viewing All Jobs
The Jobs page lists the jobs that are run for the last 30 days by default. This page lists the jobs that have been recorded in a project. This page also lists all the job types and it shows the current status of a job when it is being executed, canceled, failed, or completed.
To open the Jobs page, select the Jobs icon (
) in the left sidebar.
By default, the jobs are filtered by UI, External Tools job types. You can filter on values in the following fields:
- Search field for searching on the text of any of the visible attributes. You can also search on text in SQL statements.
- Start Time
- Status
- Job type
- User
Viewing Job Details
To view details of a specific job, perform the following steps:
On the Jobs page, click a job that you want to see the details of. The details page for the job replaces the list of jobs.
The Open Results link is on the top-right of the right panel that you can click to see the results. As long as the engine that ran the job is up, the Open Results link is visible in the UI. It disappears when the engine that ran the job shuts down and is only visible for the jobs that are run through the UI. In addition to Summary, you see the following details.
Query Fields
- Parents: Shows the dataset name and its lineage.
- Accelerated by: The reflection that accelerated the query.
- Input: Input size in bytes and the number of input records.
- Output: Output size in bytes and the number of output records.
SQL: The SQL query for the selected job.
(Optional) Click the Details tab to see the state durations, sources read, time for reading, memory used for operations, and metadata of the output.
The Details tab shows the following attributes:
State Durations: Represents the job state durations in seconds in the order they occur. Only the duration of the Engine Start state is in minutes and seconds. If the engine is stopped, it takes time to restart for the executors to be active. If the engine is already started, then Engine Start duration does not have a value.
Read Fields
- Parallelism: Number of parallel threads on the executors.
- Average Wait on Source
- Read: Memory read for this query
- Records Read
Process Fields
- Execution Time
- Top Operations: Lists the memory consumed by top operations in the query execution process
Return Fields
- Wait on Client
- Data Volume
- Number of Records
(Optional) Click the Acceleration tab to see the reflection that accelerated this job. There is an orange fire icon next to the job status indicating that the job is accelerated by a reflection.
(Optional) Click the Profile tab to see more granular details about the job.