On this page

    Reflection and Reflection-Summary Objects

    When you use Dremio’s REST API for creating, viewing, and managing reflections, you work with two types of JSON objects: reflection objects and reflection-summary objects.

    Reflection objects

    For these objects, the value of the entityType field is reflection. They look like this:

    {
        "id": String,
        "type": String ["AGGREGATION", "RAW"],
        "name": String,
        "tag": String,
        "createdAt": String (RFC3339 date),
        "updatedAt": String (RFC3339 date),
        "datasetId": String,
        "currentSizeBytes": Number,
        "totalSizeBytes": Number,
        "enabled": Boolean,
        "arrowCachingEnabled": Boolean,
        "status": {
            "config": String ["OK", "INVALID"],
            "refresh": String ["SCHEDULED", "RUNNING", "GIVEN_UP"],
            "availability": String ["NONE", "INCOMPLETE", "EXPIRED", "AVAILABLE"],    
            "combinedStatus": String ["NONE", "CAN_ACCELERATE", "CAN_ACCELERATE_WITH_FAILURES", "REFRESHING", "FAILED", "EXPIRED", "DISABLED", "INVALID", "INCOMPLETE", "CANNOT_ACCELERATE_SCHEDULED", "CANNOT_ACCELERATE_MANUAL"],
            "failureCount": Number,
            "lastDataFetch": String (RFC3339 date),
            "expiresAt": String (RFC3339 date)
        },
        "displayFields": [List of one or more fields] [valid only for reflections of type RAW],
        "dimensionFields": [List of one or more fields, including their granularity] [valid only for reflections of type AGGREGATION],
        "measureFields": [List of one or more measure fields, including their types] [valid only for reflections of type AGGREGATION],
        "distributionFields": [List of one or more fields] [optional],
        "partitionFields": [List of one or more fields] [optional],
        "sortFields": [List of one or more fields] [optional],
        "partitionDistributionStrategy": String ["CONSOLIDATED", "STRIPED"],
        "canView": Boolean,
        "canAlter": Boolean,
        "entityType": "reflection"
    }
    

    See “Descriptions of the fields” for further details.

    Example of an aggregation reflection

    Here is an example of a reflection object for an aggregation reflection:

    {
        "id": "ec919f19-3d5f-43b3-8950-093caade547d",
        "type": "AGGREGATION",
        "name": "Aggregation Reflection",
        "tag": "LmqQtmBr/U8=",
        "createdAt": "2021-10-25T06:37:57.660Z",
        "updatedAt": "2021-10-25T06:37:57.660Z",
        "datasetId": "901eff87-4741-42ba-8e59-bf7c209f0359",
        "currentSizeBytes": 11055,
        "totalSizeBytes": 22110,
        "enabled": true,
        "arrowCachingEnabled": false,
        "status": {
            "config": "OK",
            "refresh": "SCHEDULED",
            "availability": "AVAILABLE",
            "combinedStatus": "CAN_ACCELERATE",
            "failureCount": 0,
            "lastDataFetch": "2021-11-01T12:51:28.627Z",
            "expiresAt": "2021-11-01T21:51:28.627Z"
        },
        "dimensionFields": [
            {
                "name": "type",
                "granularity": "DATE"
            }
        ],
        "measureFields": [
            {
                "name": "review_count",
                "measureTypeList": [
                    "COUNT",
                    "SUM"
                ]
            },
            {
                "name": "average_stars",
                "measureTypeList": [
                    "COUNT",
                    "SUM"
                ]
            },
            {
                "name": "fans",
                "measureTypeList": [
                    "COUNT",
                    "SUM"
                ]
            },
            {
                "name": "stars",
                "measureTypeList": [
                    "COUNT",
                    "SUM"
                ]
            }
        ],
        "partitionDistributionStrategy": "CONSOLIDATED",
        "canView": true,
        "canAlter": true,
        "entityType": "reflection"
    }
    

    Example of a raw reflection

    Here is an example of a reflection object for a raw reflection:

    {
        "id": "a28fb7ea-8651-449c-80f8-3b93ae3d6032",
        "type": "RAW",
        "name": "Raw Reflection",
        "tag": "xLx2OPWQmlw=",
        "createdAt": "2021-10-22T10:41:05.509Z",
        "updatedAt": "2021-10-22T10:41:05.509Z",
        "datasetId": "a3d5e099-5f51-4a0e-b960-9865bb0651dd",
        "currentSizeBytes": 0,
        "totalSizeBytes": 0,
        "enabled": true,
        "arrowCachingEnabled": false,
        "status": {
            "config": "OK",
            "refresh": "SCHEDULED",
            "availability": "NONE",
            "combinedStatus": "CANNOT_ACCELERATE_SCHEDULED",
            "failureCount": 0,
            "lastDataFetch": "1969-12-31T23:59:59.999Z",
            "expiresAt": "1969-12-31T23:59:59.999Z"
        },
        "displayFields": [
            {
                "name": "id"
            },
            {
                "name": "Lon"
            },
            {
                "name": "Lat"
            },
            {
                "name": "zip"
            }
        ],
        "partitionDistributionStrategy": "CONSOLIDATED",
        "canView": true,
        "canAlter": true,
    }
    

    Reflection Summary

    The JSON object reflection-summary looks like this:

    {
        "id": String ,
        "type": String ["AGGREGATION", "RAW"] ,
        "name": String,
        "createdAt": String (RFC3339 date) ,
        "updatedAt": String (RFC3339 date) ,
        "currentSizeBytes": Number ,
        "totalSizeBytes": Number ,
        "enabled": Boolean,
        "arrowCachingEnabled": Boolean,
        "datasetId": String,
        "datasetType": String ["PHYSICAL_DATASET", "VIRTUAL_DATASET"],
        "datasetPath": [String],
        "status": {
            "config": String ["OK", "INVALID"] ,
            "refresh": String ["SCHEDULED", "RUNNING", "GIVEN_UP"] ,
            "availability": String ["NONE", "INCOMPLETE", "EXPIRED", "AVAILABLE"] ,    
            "combinedStatus": String ["NONE", "CAN_ACCELERATE", "CAN_ACCELERATE_WITH_FAILURES", "REFRESHING", "FAILED", "EXPIRED", "DISABLED", "INVALID", "INCOMPLETE", "CANNOT_ACCELERATE_SCHEDULED", "CANNOT_ACCELERATE_MANUAL"] ,
            "failureCount": Number ,
            "lastDataFetch": String (RFC3339 date) ,
            "expiresAt": String (RFC3339 date) 
        },
        "canView": Boolean,
        "canAlter": Boolean,
        "entityType": "reflection-summary" 
    }
    

    See “Descriptions of the fields” for further details.

    Differences between reflection objects and reflection-summary objects

    A reflection-summary differs from a reflection in these ways:

    • It lacks the tag field.
    • It moves the datasetId field.
    • It includes the datasetType and datasetPath fields.
    • It lacks these fields:
      • distributionFields
      • partitionFields
      • sortFields
      • partitionDistributionStrategy
    • The value for the entity field is reflection-summary.

    A reflection-summary for a raw reflection also lacks the field displayFields. For an aggregation reflection, it lacks the fields dimensionFields and measureFields.

    This image shows the differences between a reflection object, which is on the left, and a reflection-summary object, which is on the right:

    Reflection versus Reflection Summary

    Descriptions of the fields

    FieldTypeDescription
    idStringReflection ID. The value is generated by Dremio and is immutable.
    typeStringThe reflection type, must either be AGGREGATION or RAW. This value cannot be changed after a reflection is created.
    nameStringName of the reflection, required to be non empty.
    tagStringIdentifies the instance of the reflection, changed each time a source is modified. The value is generated by Dremio and is immutable.
    createdAtStringRFC3339 date (example: 2017-10-27T21:08:22.858Z) representing the creation datetime. The value is generated by Dremio and is immutable.
    updatedAtStringRFC3339 date (example: 2017-10-27T21:08:22.858Z) representing the last time the reflection was updated. The value is generated by Dremio and is immutable.
    datasetIdStringThe id of the dataset the reflection is for. Immutable after creation.
    currentSizeBytesNumberThe data size (in bytes) of the latest reflection job (if one exists). The value is generated by Dremio and is immutable.
    totalSizeBytesNumberThe data size (in bytes) of all reflection jobs that have not been pruned (if any exist). The value is generated by Dremio and is immutable.
    enabledBooleanWhether to allow using the reflection to accelerate queries.
    arrowCachingEnabledBooleanWhether Dremio converts data from your reflection’s Parquet files to the Apache Arrow format when copying that data to executor nodes.
    statusListStatus of the reflection. See “Reflection Status” below.
    dimensionFieldsListList of dimension fields, of format {"name": "string", "granularity": ["NORMAL", "DATE"]}. Valid only for AGGREGATION reflections.
    measureFieldsListList of measure fields, of format {"name": "string", "measureTypeList": ["MIN", "MAX", "SUM", "COUNT", "APPROX_COUNT_DISTINCT"]}. The default is SUM and COUNT, and the SUM value is not valid for text and date fields. Valid only for AGGREGATION reflections.
    displayFieldsListList of fields, of format {"name": "string"}, included from the anchor dataset. Valid only for RAW reflections.
    sortFieldsListOptional list of sort fields, of format {"name": "string"}.
    partitionFieldsListOptional list of partition fields, of format {"name": "string"}.
    distributionFieldsListOptional list of distribution fields, of format {"name": "string"}.
    partitionDistributionStrategyStringMust either be CONSOLIDATED or STRIPED.

    Description of the subfields of the status field

    FieldTypeDescription
    configStringState of the reflection configuration. Either OK or INVALID (the dataset schema has changed and the reflection definition is no longer applicable, for example a field that was being used no longer exists). The value is generated by Dremio and is immutable.
    refreshStringSee “Descriptions of values for refresh” below for values and their descriptions. The value is generated by Dremio and is immutable.
    availablityStringWhether the reflection is available to accelerate queries. Either NONE, INCOMPLETE, EXPIRED or AVAILABLE. The value is generated by Dremio and is immutable.
    combinedStatusStringSee “Descriptions of values for combinedStatus” below for values and their descriptions. The value is generated by Dremio and is immutable.
    failureCountNumberNumber of consecutive reflection creation failures. The value is generated by Dremio and is immutable.
    lastDataFetchStringRFC3339 date (example: 2017-10-27T21:08:22.858Z) representing the last time the reflection data was refreshed. The value is generated by Dremio and is immutable.
    expiresAtStringRFC3339 date (example: 2017-10-27T21:08:22.858Z) representing when the reflection data will expire. The value is generated by Dremio and is immutable.

    Descriptions of values for refresh

    These are the possible values of the refresh field:

    StatusDescription
    GIVEN_UPDremio has attempted multiple times to refresh the reflection, but each attempt has failed. Dremio will not make further attempts.
    MANUALThe refresh period is set to 0, so refreshes of the reflection must be triggered manually from within Dremio's user interface.
    RUNNINGDremio is in the process of refreshing the reflection.
    SCHEDULEDRefreshes of the reflection occur according to a schedule.

    Descriptions of values for combinedStatus

    These are the possible values of the combinedStatus field. These values also appear in the status column of the results for the query select * from sys.reflections.

    StatusDescription
    CAN_ACCELERATEReflection is in fully functional.
    CAN_ACCELERATE_WITH_FAILURESThe last refresh failed to obtain a status, but Dremio still has a valid materialization.
    CANNOT_ACCELERATE_MANUALReflection is currently unable to accelerate any queries and has the option Never Refresh selected for the Refresh Policy.
    CANNOT_ACCELERATE_SCHEDULEDReflection is currently unable to accelerate any queries, but it has been scheduled for a refresh at a future time.
    DISABLEDReflection has been manually disabled in Dremio.
    EXPIREDReflection has expired and cannot be used.
    FAILEDReflection has failed in its attempt to refresh. This typically indicates that three (3) subsequent refresh attempts have failed. It is still usable.
    INCOMPLETEOne or more PDFS nodes that contain materialized files are down. This should not happen for configurations using Hadoop Distributed File System (HDFS) to store reflections. Only partial data is available.
    INVALIDReflection’s layout has an invalid configuration. This means that the underlying dataset has changed.
    REFRESHINGReflection is currently refreshing its materialization.