Reflection

This topic covers the Reflections functionality in Dremio. All reflections returned by the REST API will have a _type of reflection.

JSON representation of a reflection

{
  "entityType": "reflection" [immutable, generated by Dremio],
  "id": String [immutable, generated by Dremio],
  "tag": String [immutable, generated by Dremio],
  "name": String,
  "enabled": Boolean,
  "createdAt": String (RFC3339 date) [immutable, generated by Dremio],
  "updatedAt": String (RFC3339 date) [immutable, generated by Dremio],
  "type": String ["AGGREGATION", "RAW"] [immutable after creation],
  "datasetId": String [immutable after creation],
  "currentSizeBytes": Number [immutable, generated by Dremio],
  "totalSizeBytes": Number [immutable, generated by Dremio],
  "status": Reflection Status [immutable, generated by Dremio],
  "dimensionFields": [Reflection Field With Granularity] [optional, only valid for AGGREGATION],
  "measureFields": [Reflection Field] [optional, only valid for AGGREGATION],
  "displayFields": [Reflection Field] [optional, only valid for RAW],
  "distributionFields": [Reflection Field] [optional],
  "partitionFields": [Reflection Field] [optional],
  "sortFields": [Reflection Field] [optional],
  "partitionDistributionStrategy": String ["CONSOLIDATED", "STRIPED"],
}

Parameter Description

Name Type Description
id String Reflection ID. Generated by Dremio, immutable.
tag String Identifies the instance of the reflection, changed each time a source is modified. Generated by Dremio, immutable.
name String Name of the reflection, required to be non empty.
enabled Boolean Whether to allow using the reflection to accelerate queries.
createdAt String RFC3339 date (example: 2017-10-27T21:08:22.858Z) representing the creation datetime. Generated by Dremio, immutable.
updatedAt String RFC3339 date (example: 2017-10-27T21:08:22.858Z) representing the last time the reflection was updated. Generated by Dremio, immutable.
type String The reflection type, must either be AGGREGATION or RAW. Immutable after creation.
datasetId String The id of the dataset the reflection is for. Immutable after creation.
currentSizeBytes Number The data size (in bytes) of the latest reflection job (if one exists). Generated by Dremio, immutable.
totalSizeBytes Number The data size (in bytes) of all reflection jobs that have not been pruned (if any exist). Generated by Dremio, immutable.
status [Reflection Status] Status of the reflection. Generated by Dremio, immutable.
dimensionFields Reflection Field With Granularity[] List of dimension fields, optional and only valid for AGGREGATION reflections.
measureFields Reflection Measure Field[] List of measure fields, optional and only valid for AGGREGATION reflections.
displayFields Reflection Field[] List of display fields, optional and only valid for RAW reflections.
sortFields Reflection Field[] List of sort fields, optional and valid for all reflections.
partitionFields Reflection Field[] List of partition fields, optional and valid for all reflections.
distributionFields Reflection Field[] List of distribution fields, optional and valid for all reflections.
partitionDistributionStrategy String Must either be CONSOLIDATED or STRIPED.

Reflection Field

{
  "name": String
}

Reflection Field With Granularity

{
  "name": String,
  "granularity": String ["NORMAL", "DATE"]
}

Reflection Measure Field

This section discusses a measure field.

The measureTypeList field was introduced in Dremio 2.1.0. It represents the list of statistics that will be pre-computed. The default is SUM and COUNT and the SUM value is not valid for text and date columns.

{
  "name": String,
  "measureTypeList": String[]  ['MIN', 'MAX', 'SUM', 'COUNT', 'APPROX_COUNT_DISTINCT']
}

Reflection Status

This section illustrates the status of a reflection.

{
  "config": String ["OK", "INVALID"] [immutable, generated by Dremio],
  "refresh": String ["SCHEDULED", "RUNNING", "GIVEN_UP"] [immutable, generated by Dremio],
  "availability": String ["NONE", "INCOMPLETE", "EXPIRED", "AVAILABLE"] [immutable, generated by Dremio],
  "failureCount": Number [immutable, generated by Dremio],
  "lastRefresh": String (RFC3339 date) [immutable, generated by Dremio],
  "expiresAt": String (RFC3339 date) [immutable, generated by Dremio]
}
Name Type Description
config String State of the reflection configuration. Either OK or INVALID (the dataset schema has changed and the reflection definition is no longer applicable, for example a field that was being used no longer exists). Generated by Dremio, immutable.
refresh String Refresh state of the reflection. Either MANUAL (refresh period is set to 0 and user must manually refresh), SCHEDULED, RUNNING or GIVEN_UP (the reflection has failed to refresh multiple times and will not be reattempted). Generated by Dremio, immutable.
availablity String Whether the reflection is available to accelerate queries. Either NONE, INCOMPLETE, EXPIRED or AVAILABLE. Generated by Dremio, immutable.
failureCount Number Number of consecutive reflection creation failures. Generated by Dremio, immutable.
lastRefresh String RFC3339 date (example: 2017-10-27T21:08:22.858Z) representing the last time the reflection data was refreshed. Generated by Dremio, immutable.
expiresAt String RFC3339 date (example: 2017-10-27T21:08:22.858Z) representing when the reflection data will expire. Generated by Dremio, immutable.
Refresh Statuses

This section outlines all the states of the reflection that appear for refresh.

The following are statuses shown next to Jobs after navigating in Dremio to Administrator > Reflections.

Icon Status Description
Active The reflection is in a good condition and may be used for queries presently.
Disabled The reflection has been disabled or is no longer available for use. The reflection materialization is stored and queries using this reflection are disabled. If failed, this means that three (3) subsequent refresh attempts have failed.
Refreshing The reflection is currently refreshing.
In-Progress/Running Reflection is presently active or in use for ongoing queries.
Warning A refresh attempt has failed, but the reflection has not yet expired. The reflection may be in the midst of a refresh.

The following are statuses that appear when users perform a query using select * from sys.reflections. This shows the internal state of Reflection.

Status Description
CAN_ACCELERATE Reflection is in fully functional.
CAN_ACCELERATE_WITH_FAILURES The last refresh failed to obtain a status, but Dremio still has a valid materialization.
REFRESHING Reflection is currently refreshing its materialization.
FAILED Reflection has failed in its attempt to refresh. This typically indicates that three (3) subsequent refresh attempts have failed. It is still usable.
EXPIRED Reflection has expired and cannot be used.
DISABLED Reflection has been manually disabled in Dremio.
INVALID Reflection’s layout has an invalid configuration. This means that the underlying dataset has changed.
INCOMPLETE One or more PDF nodes that contain materialized files are down. This should not happen for configurations using Hadoop Distributed File System (HDFS) to store reflections. Only partial data is available.
CANNOT_ACCELERATE_SCHEDULED Reflection is currently unable to accelerate any queries, but it has been scheduled for a refresh at a future time.
CANNOT_ACCELERATE_MANUAL Reflection is currently unable to accelerate any queries and has the option Never Refresh selected for the Refresh Policy.

Reflection Summary

This section covers the summary of a reflection in Dremio. Includes a subset of reflection with some additional dataset information.

The JSON representation of a reflection summary looks like this:

{
  "_type": "reflection-summary" [immutable],
  "id": String [immutable],
  "createdAt": String (RFC3339 date) [immutable],
  "updatedAt": String (RFC3339 date) [immutable],
  "type": String ["AGGREGATION", "RAW"] [immutable],
  "name": String [immutable],
  "datasetId": String [immutable],
  "datasetPath": [String] [immutable],
  "datasetType": String ["PHYSICAL", "VIRTUAL"] [immutable],
  "currentSizeBytes": Number [immutable],
  "totalSizeBytes": Number [immutable],
  "enabled": Boolean [immutable],
  "status": Reflection Status [immutable]
}