This topic describes job profiles.
A profile is a summary of metrics collected for each executed query. Query profiles provide information that can be used to monitor and analyze query performance. These profile metrics are available via the Dremio UI Jobs > Profile.
Within the Dremio UI Jobs Profile tab, you can display information based on the following views:
- Visualized Plan
- Error (if applicable)
Each view displays the following metrics information: (Query, Visualized Plan, Planning, and Acceleration).
- Resource Allocation
The Query view shows the selected query statement and a job summary along with job metrics information. The job summary information includes:
- Query Text
- Job State: Completed, Failed, Cancelled
- Name of Coordinator
- Command Pool Wait Time
- Planning Time
- Resource Scheduling Time
The Visualized Plan view shows a visualized diagram and a job summary along with job metrics information. The Job profile visualized plan is useful in understanding the flow of the query.
[info] The detailed visualized pan diagram is always read from the bottom up.
The Planning view shows planning metrics, query output schema, non default options, and a job summary along with job metrics information.
The Planning view provides statistics about the actual cost of the query operations in terms of memory, I/O, and CPU processing. You can use this profile to identify which operations consumed the majority of the resources during a query, modify the physical plan to address the cost-intensive operations. In particular, the following information is useful:
- Non Default Options
- Metadata Cache Hits and Misses with times
- Final Physical Transformation - For example, look for push down query for RDBMS, MongoDB, or ElasticSearch, filter pushdowns or partition pruning for Parquet, the usage of stripes for ORC and so on.
- Compare estimated row count versus the actual scan, join, or aggregate result.
The following screenshot shows some initial planning information.
The following screenshot shows Query Output Schema, Non Default Options, and Job Summary information.
The Acceleration view shows reflection outcome, canonicalized user query alternatives, reflection details, and a job summary along with job metrics information.
The following considerations determines the acceleration process:
- Considered, Matched, Chosen -- The query is accelerated.
- Considered, Matched, Not Chosen -- The query is not accelerated because either a costing issue or an exception during substitution occurred.
- Considered, Not Matched, Not Chosen -- The query is not accelerated because the reflection does not have the data to accelerate.
The following screenshot shows some basic information.
The Error view shows information about an errors.
- Failure Node -- This node is always the coordinator node.
- Server name inside the error message is the actual affected node
For More Information
- Job Metrics
- Downloading Profiles
- Uploading Profile Data
- Querying Profile Datasets
- Analyzing Profiles