On this page

    Viewing Jobs and Job Details

    Dremio Cloud performs these types of task through jobs:

    • Refreshing metadata
    • Creating, revising, refreshing, and removing reflections
    • Running queries submitted through the Dremio UI, JDBC, ODBC, or REST APIs

    Job Attributes

    Each job has the following attributes, which can appear in columns in the list of jobs:

    (Accelerated)
    The heading for this column is a purple lightning bolt. A lightning bolt in a row indicates that the query that the job ran was accelerated by one or more reflections.
    Dataset
    The dataset queried, if one was queried.
    Duration
    The length of time (in seconds) that a job required from start to completion.
    Job ID
    A universally unique identifier.
    Planner Cost Estimate
    A cost estimate calculated by Dremio based on an evaluation of the resources that to be used in the execution of a query. The number is not in units, and is intended to give a an idea of the cost of executing a query relative to the costs of executing other queries. Values are derived by adding weighted estimates of required I/O, memory, and CPU load. In reported values, K = thousand, M = million, B = billion, and T = trillion. For example, a value of 12,543,765,321 is reported as 12.5B.
    Planning Time
    The length of time (in seconds) in which the query optimizer planned the execution of the query.
    Query Type
    Represents the query type. For descriptions of the types, see Job Types.
    Start Time
    The date and time at which the job began.
    (Status)
    An icon that represents the current status of the job. A status represents one or more job states. For descriptions, see Job States and Statuses.
    User
    Username of the user who ran the query and initiated the job.

    Job States and Statuses

    Each job passes through a sequence of states until it is complete, though the sequence can be interrupted if a query is cancelled or if there is an error during a state. In this diagram, the states that a job passes through are in white, and the possible end states are in dark gray.

    StateDescription
    PendingRepresents a state where the query is waiting to be scheduled on the query pool.
    Metadata RetrievalRepresents a state where metadata schema is retrieved and the SQL command is parsed.
    PlanningRepresents a state where the following are done:
    • Physical and logical planning
    • Reflection matching
    • Partition metadata retrieval
    • Mapping the query to a queue based workload management rules
    • Pick the engine associated with the query to run the query.
    Engine StartRepresents a state where the engine starts if it has stopped. If the engine is stopped, it takes time to restart for the executors to be active. If the engine is already started, then this state does not have a duration.
    QueuedRepresents a state where a job is queued. Each queue has a limit of concurrent queries. If the queries in progress exceed the concurrency limit, the query should wait in the queue until the jobs in progress complete.
    Execution PlanningRepresents a state where executor nodes are selected from the chosen engine to run the query, and work is distributed to each executor.
    RunningRepresents a state where executor nodes execute and complete the fragments assigned to them. Typically, most queries spend more time in this state.
    CanceledRepresents a terminal state that indicates that the query is canceled by the user or an intervention in the system.
    CompletedRepresents a terminal state that indicates that the query is successfully completed.
    FailedRepresents a terminal state that indicates that the query has failed due to an error.

    This table lists the statuses that the UI lets you filter on and shows how they map to the states:

    StatusState
    SetupPending
    Metadata Retrieval
    Planning
    Engine StartEngine Start
    QueuedQueued
    RunningExecution Planning
    Starting
    Running
    CanceledCanceled
    CompletedCompleted
    FailedFailed

    Job Types

    There are five types of jobs:

    TypeDescription
    AcceleratorJobs related to creating, maintaining, and removing reflections.
    DownloadsJobs that are used to download datasets.
    External ToolsJobs due to queries from client applications, such as Microsoft Power BI, Superset, Tableau, other third-party client applications, and custom applications.
    InternalJobs for queries that Dremio Cloud submits for internal operations.
    UIJobs due to queries issued from the SQL Runner in the Dremio Cloud UI.

    Viewing All Jobs

    The Jobs page lists the jobs that are run for the last 30 days by default. This page lists the jobs that have been recorded in a project. This page also lists all the job types and it shows the current status of a job when it is being executed, canceled, failed, or completed.

    To open the Jobs page, select the Jobs icon () in the left sidebar.

    By default, the jobs are filtered by UI, External Tools job types. You can filter on values in the following fields:

    • Search field for searching on the text of any of the visible attributes. You can also search on text in SQL statements.
    • Start Time
    • Status
    • Job type
    • User

    Viewing Job Details

    To view details of a specific job, perform the following steps:

    1. On the Jobs page, click a job that you want to see the details of. The details page for the job replaces the list of jobs.

      The Open Results link is on the top-right of the right panel that you can click to see the results. As long as the engine that ran the job is up, the Open Results link is visible in the UI. It disappears when the engine that ran the job shuts down and is only visible for the jobs that are run through the UI. In addition to Summary, you see the following details.

      • Query Fields

        • Parents: Shows the dataset name and its lineage.
        • Accelerated by: The reflection that accelerated the query.
        • Input: Input size in bytes and the number of input records.
        • Output: Output size in bytes and the number of output records.
      • SQL: The SQL query for the selected job.

    2. (Optional) Click the Details tab to see the state durations, sources read, time for reading, memory used for operations, and metadata of the output.

      The Details tab shows the following attributes:

      • State Durations: Represents the job state durations in seconds in the order they occur. Only the duration of the Engine Start state is in minutes and seconds. If the engine is stopped, it takes time to restart for the executors to be active. If the engine is already started, then Engine Start duration does not have a value.

      • Read Fields

        • Parallelism: Number of parallel threads on the executors.
        • Average Wait on Source
        • Read: Memory read for this query
        • Records Read
      • Process Fields

        • Execution Time
        • Top Operations: Lists the memory consumed by top operations in the query execution process
      • Return Fields

        • Wait on Client
        • Data Volume
        • Number of Records
    3. (Optional) Click the Acceleration tab to see the reflection that accelerated this job. There is an orange fire icon next to the job status indicating that the job is accelerated by a reflection.

    4. (Optional) Click the Profile tab to see more granular details about the job.