Reflection and Reflection-Summary Objects

When you use Dremio’s REST API for creating, viewing, and managing reflections, you work with two types of JSON objects: reflection objects and reflection-summary objects.

Reflection objects

For these objects, the value of the entityType field is reflection. They look like this:

{
    "id": String,
    "type": String ["AGGREGATION", "RAW"],
    "name": String,
    "tag": String,
    "createdAt": String (RFC3339 date),
    "updatedAt": String (RFC3339 date),
    "datasetId": String,
    "currentSizeBytes": Number,
    "totalSizeBytes": Number,
    "enabled": Boolean,
    "arrowCachingEnabled": Boolean,
    "status": {
        "config": String ["OK", "INVALID"],
        "refresh": String ["SCHEDULED", "RUNNING", "GIVEN_UP"],
        "availability": String ["NONE", "INCOMPLETE", "EXPIRED", "AVAILABLE"],    
        "combinedStatus": String ["NONE", "CAN_ACCELERATE", "CAN_ACCELERATE_WITH_FAILURES", "REFRESHING", "FAILED", "EXPIRED", "DISABLED", "INVALID", "INCOMPLETE", "CANNOT_ACCELERATE_SCHEDULED", "CANNOT_ACCELERATE_MANUAL"],
        "failureCount": Number,
        "lastDataFetch": String (RFC3339 date),
        "expiresAt": String (RFC3339 date)
    },
    "displayFields": [List of one or more fields] [valid only for reflections of type RAW],
    "dimensionFields": [List of one or more fields, including their granularity] [valid only for reflections of type AGGREGATION],
    "measureFields": [List of one or more measure fields, including their types] [valid only for reflections of type AGGREGATION],
    "distributionFields": [List of one or more fields] [optional],
    "partitionFields": [List of one or more fields] [optional],
    "sortFields": [List of one or more fields] [optional],
    "partitionDistributionStrategy": String ["CONSOLIDATED", "STRIPED"],
    "canView": Boolean,
    "canAlter": Boolean,
    "entityType": "reflection"
}

See “Descriptions of the fields” for further details.

Example of an aggregation reflection

Here is an example of a reflection object for an aggregation reflection:

{
    "id": "ec919f19-3d5f-43b3-8950-093caade547d",
    "type": "AGGREGATION",
    "name": "Aggregation Reflection",
    "tag": "LmqQtmBr/U8=",
    "createdAt": "2021-10-25T06:37:57.660Z",
    "updatedAt": "2021-10-25T06:37:57.660Z",
    "datasetId": "901eff87-4741-42ba-8e59-bf7c209f0359",
    "currentSizeBytes": 11055,
    "totalSizeBytes": 22110,
    "enabled": true,
    "arrowCachingEnabled": false,
    "status": {
        "config": "OK",
        "refresh": "SCHEDULED",
        "availability": "AVAILABLE",
        "combinedStatus": "CAN_ACCELERATE",
        "failureCount": 0,
        "lastDataFetch": "2021-11-01T12:51:28.627Z",
        "expiresAt": "2021-11-01T21:51:28.627Z"
    },
    "dimensionFields": [
        {
            "name": "type",
            "granularity": "DATE"
        }
    ],
    "measureFields": [
        {
            "name": "review_count",
            "measureTypeList": [
                "COUNT",
                "SUM"
            ]
        },
        {
            "name": "average_stars",
            "measureTypeList": [
                "COUNT",
                "SUM"
            ]
        },
        {
            "name": "fans",
            "measureTypeList": [
                "COUNT",
                "SUM"
            ]
        },
        {
            "name": "stars",
            "measureTypeList": [
                "COUNT",
                "SUM"
            ]
        }
    ],
    "partitionDistributionStrategy": "CONSOLIDATED",
    "canView": true,
    "canAlter": true,
    "entityType": "reflection"
}

Example of a raw reflection

Here is an example of a reflection object for a raw reflection:

{
    "id": "a28fb7ea-8651-449c-80f8-3b93ae3d6032",
    "type": "RAW",
    "name": "Raw Reflection",
    "tag": "xLx2OPWQmlw=",
    "createdAt": "2021-10-22T10:41:05.509Z",
    "updatedAt": "2021-10-22T10:41:05.509Z",
    "datasetId": "a3d5e099-5f51-4a0e-b960-9865bb0651dd",
    "currentSizeBytes": 0,
    "totalSizeBytes": 0,
    "enabled": true,
    "arrowCachingEnabled": false,
    "status": {
        "config": "OK",
        "refresh": "SCHEDULED",
        "availability": "NONE",
        "combinedStatus": "CANNOT_ACCELERATE_SCHEDULED",
        "failureCount": 0,
        "lastDataFetch": "1969-12-31T23:59:59.999Z",
        "expiresAt": "1969-12-31T23:59:59.999Z"
    },
    "displayFields": [
        {
            "name": "id"
        },
        {
            "name": "Lon"
        },
        {
            "name": "Lat"
        },
        {
            "name": "zip"
        }
    ],
    "partitionDistributionStrategy": "CONSOLIDATED",
    "canView": true,
    "canAlter": true,
}

Reflection Summary

The JSON object reflection-summary looks like this:

{
    "id": String ,
    "type": String ["AGGREGATION", "RAW"] ,
    "name": String,
    "createdAt": String (RFC3339 date) ,
    "updatedAt": String (RFC3339 date) ,
    "currentSizeBytes": Number ,
    "totalSizeBytes": Number ,
    "enabled": Boolean,
    "arrowCachingEnabled": Boolean,
    "datasetId": String,
    "datasetType": String ["PHYSICAL_DATASET", "VIRTUAL_DATASET"],
    "datasetPath": [String],
    "status": {
        "config": String ["OK", "INVALID"] ,
        "refresh": String ["SCHEDULED", "RUNNING", "GIVEN_UP"] ,
        "availability": String ["NONE", "INCOMPLETE", "EXPIRED", "AVAILABLE"] ,    
        "combinedStatus": String ["NONE", "CAN_ACCELERATE", "CAN_ACCELERATE_WITH_FAILURES", "REFRESHING", "FAILED", "EXPIRED", "DISABLED", "INVALID", "INCOMPLETE", "CANNOT_ACCELERATE_SCHEDULED", "CANNOT_ACCELERATE_MANUAL"] ,
        "failureCount": Number ,
        "lastDataFetch": String (RFC3339 date) ,
        "expiresAt": String (RFC3339 date) 
    },
    "canView": Boolean,
    "canAlter": Boolean,
    "entityType": "reflection-summary" 
}

See “Descriptions of the fields” for further details.

Differences between reflection objects and reflection-summary objects

A reflection-summary differs from a reflection in these ways:

  • It lacks the tag field.
  • It moves the datasetId field.
  • It includes the datasetType and datasetPath fields.
  • It lacks these fields:
    • distributionFields
    • partitionFields
    • sortFields
    • partitionDistributionStrategy
  • The value for the entity field is reflection-summary.

A reflection-summary for a raw reflection also lacks the field displayFields. For an aggregation reflection, it lacks the fields dimensionFields and measureFields.

This image shows the differences between a reflection object, which is on the left, and a reflection-summary object, which is on the right:

Reflection versus Reflection Summary

Descriptions of the fields

Field Type Description
id String Reflection ID. The value is generated by Dremio and is immutable.
type String The reflection type, must either be AGGREGATION or RAW. This value cannot be changed after a reflection is created.
name String Name of the reflection, required to be non empty.
tag String Identifies the instance of the reflection, changed each time a source is modified. The value is generated by Dremio and is immutable.
createdAt String RFC3339 date (example: 2017-10-27T21:08:22.858Z) representing the creation datetime. The value is generated by Dremio and is immutable.
updatedAt String RFC3339 date (example: 2017-10-27T21:08:22.858Z) representing the last time the reflection was updated. The value is generated by Dremio and is immutable.
datasetId String The id of the dataset the reflection is for. Immutable after creation.
currentSizeBytes Number The data size (in bytes) of the latest reflection job (if one exists). The value is generated by Dremio and is immutable.
totalSizeBytes Number The data size (in bytes) of all reflection jobs that have not been pruned (if any exist). The value is generated by Dremio and is immutable.
enabled Boolean Whether to allow using the reflection to accelerate queries.
arrowCachingEnabled Boolean Whether Dremio converts data from your reflection’s Parquet files to the Apache Arrow format when copying that data to executor nodes.
status List Status of the reflection. See “Reflection Status” below.
dimensionFields List List of dimension fields, of format {"name": "string", "granularity": ["NORMAL", "DATE"]}. Valid only for AGGREGATION reflections.
measureFields List List of measure fields, of format {"name": "string", "measureTypeList": ["MIN", "MAX", "SUM", "COUNT", "APPROX_COUNT_DISTINCT"]}. The default is SUM and COUNT, and the SUM value is not valid for text and date fields. Valid only for AGGREGATION reflections.
displayFields List List of fields, of format {"name": "string"}, included from the anchor dataset. Valid only for RAW reflections.
sortFields List Optional list of sort fields, of format {"name": "string"}.
partitionFields List Optional list of partition fields, of format {"name": "string"}.
distributionFields List Optional list of distribution fields, of format {"name": "string"}.
partitionDistributionStrategy String Must either be CONSOLIDATED or STRIPED.

Description of the subfields of the status field

Field Type Description
config String State of the reflection configuration. Either OK or INVALID (the dataset schema has changed and the reflection definition is no longer applicable, for example a field that was being used no longer exists). The value is generated by Dremio and is immutable.
refresh String See “Descriptions of values for refresh” below for values and their descriptions. The value is generated by Dremio and is immutable.
availablity String Whether the reflection is available to accelerate queries. Either NONE, INCOMPLETE, EXPIRED or AVAILABLE. The value is generated by Dremio and is immutable.
combinedStatus String See “Descriptions of values for combinedStatus” below for values and their descriptions. The value is generated by Dremio and is immutable.
failureCount Number Number of consecutive reflection creation failures. The value is generated by Dremio and is immutable.
lastDataFetch String RFC3339 date (example: 2017-10-27T21:08:22.858Z) representing the last time the reflection data was refreshed. The value is generated by Dremio and is immutable.
expiresAt String RFC3339 date (example: 2017-10-27T21:08:22.858Z) representing when the reflection data will expire. The value is generated by Dremio and is immutable.

Descriptions of values for refresh

These are the possible values of the refresh field:

Status Description
GIVEN_UP Dremio has attempted multiple times to refresh the reflection, but each attempt has failed. Dremio will not make further attempts.
MANUAL The refresh period is set to 0, so refreshes of the reflection must be triggered manually from within Dremio's user interface.
RUNNING Dremio is in the process of refreshing the reflection.
SCHEDULED Refreshes of the reflection occur according to a schedule.

Descriptions of values for combinedStatus

These are the possible values of the combinedStatus field. These values also appear in the status column of the results for the query select * from sys.reflections.

Status Description
CAN_ACCELERATE Reflection is in fully functional.
CAN_ACCELERATE_WITH_FAILURES The last refresh failed to obtain a status, but Dremio still has a valid materialization.
CANNOT_ACCELERATE_MANUAL Reflection is currently unable to accelerate any queries and has the option Never Refresh selected for the Refresh Policy.
CANNOT_ACCELERATE_SCHEDULED Reflection is currently unable to accelerate any queries, but it has been scheduled for a refresh at a future time.
DISABLED Reflection has been manually disabled in Dremio.
EXPIRED Reflection has expired and cannot be used.
FAILED Reflection has failed in its attempt to refresh. This typically indicates that three (3) subsequent refresh attempts have failed. It is still usable.
INCOMPLETE One or more PDF nodes that contain materialized files are down. This should not happen for configurations using Hadoop Distributed File System (HDFS) to store reflections. Only partial data is available.
INVALID Reflection’s layout has an invalid configuration. This means that the underlying dataset has changed.
REFRESHING Reflection is currently refreshing its materialization.