Job Metrics

This topic describes the job metric information (threads, resource allocation, nodes, and operators) that is displayed with each view (Query, Visualized Plan, Planning, and Acceleration).

Threads

The Thread section provides an overview table and a major fragment block for each major fragment. Each row in the Overview table provides the number of minor fragment that Dremio parallelized from each major fragment, as well as aggregate time and memory metrics for the minor fragments.

In particular, the following metrics are useful:

  • Setup Time -- Time opening and closing of files.
  • Waiting -- Time waiting on CPU.
  • Blocked on Downstream -- Represents completed work whereas the next phase is not ready to accept work.
  • Blocked on Upstream -- Represents the phase before it is ready to give work though the current phase is not ready.
  • Phase Metrics -- Displays memory used per node (Phases can run in parallel)

Major fragment blocks correspond to a row in the Overview table. You can expand the blocks to see metrics for all of the minor fragment that were parallelized from each major fragment, including the host on which each minor fragment ran. Each row in the major fragment table presents the fragment state, time metrics, memory metrics, and aggregate input metrics of each minor fragment.

Resource Allocation

Nodes

Operator Overview Table

Shows aggregate metrics for each operator within a major fragment that performed relational operations during query execution.

The following table lists descriptions for each column in the Operator Overview table:

Column Name Description
Operator ID The coordinates of an operator that performed an operation during a particular phase of the query. For example, 02-xx-03 where 02 is the major fragment ID, xx corresponds to a minor fragment ID, and 03 is the Operator ID.
Type The operator type. Operators can be of type project, filter, hash join, single sender, or unordered receiver.
Min Setup Time, Avg Setup Time, Max Setup Time In general, the time spent opening and closing files. Specifically, the minimum, average, and maximum amount of time spent by the operator to set up before performing the operation.
Min Process Time, Avg Process Time, Max Process Time In general, the time spent on acctualy processing the rows. Specifically, the minimum, average, and maximum amount of time spent by the operator to process/read records from the file system or database.
Wait (min, avg, max) In general, the time spent waiting on Disk I/O. These fields represent the minimum, average, and maximum times spent by operators waiting on disk I/O.
Avg Peak Memory Represents the average of the peak direct memory allocated across minor fragment. Relates to the memory needed by operators to perform their operations, such as hash join or sort.
Max Peak Memory Represents the maximum of the peak direct memory allocated across minor fragment. Relates to the memory needed by operators to perform their operations, such as hash join or sort.

Operator Block

Shows time and memory metrics for each operator type within a major fragment. Operator types include:

  • SCREEN
  • PROJECT
  • WRITER_COMMITTER
  • ARROW_WRITER

The following table provides descriptions for each column presented in the operator block:

Column Name Description
Thread The coordinate ID of the minor fragment on which the operator ran. For example, 04-03-01 where 04 is the major fragment ID, 03 is the minor fragment ID, and 01 is the Operator ID.
Setup Time The amount of time spent by the operator to set up before performing its operation. This includes run-time code generation and opening a file.
Process Time The amount of time spent by the operator to perform its operation.
Wait Time The cumulative amount of time spent by an operator waiting for external resources. such as waiting to send records, waiting to receive records, waiting to write to disk, and waiting to read from disk.
Max Batches The maximum number of record batches consumed from a single input stream.
Max Records The maximum number of records consumed from a single input stream.
Peak Memory Represents the peak direct memory allocated. Relates to the memory needed by the operators to perform their operations, such as hash join and sort.

When viewing operation details, the following are observable:

  • The degree of parallelism and number of minor fragments.
  • The setup, process, and wait time per minor fragments.

For example, the following sample screenshot shows a SCREEN operator block:


results matching ""

    No results matching ""