Dremio Cloud
Dremio Software
 
On this page

    Reflection Summary (23.x and earlier) enterprise

    Use the Reflection API to retrieve a reflection summary that includes the raw and aggregation reflections for the Dremio instance.

    Reflection summary objects are different from reflection objects. Reflection summaries do not include certain attributes that define the reflection, like the display, dimension, measure, sort, and partition attributes. Reflection summaries do include several attributes that do not appear in reflection objects, like datasetType and datasetPath.

    Reflection Summary Object 
    {
  "data": [
    {
      "id": "85dda9dd-2371-467f-b68d-fc4c5ea57a8b",
      "type": "AGGREGATION",
      "name": "Aggregation Reflection",
      "createdAt": "2022-07-05T19:19:40.244Z",
      "updatedAt": "2023-01-10T17:12:30.244Z",
      "currentSizeBytes": 18639885,
      "totalSizeBytes": 142639924,
      "enabled": true,
      "arrowCachingEnabled": false,
      "status": {
        "config": "OK",
        "refresh": "SCHEDULED",
        "availability": "AVAILABLE",
        "combinedStatus": "CAN_ACCELERATE",
        "failureCount": 0,
        "lastDataFetch": "2023-01-23T19:37:37.030Z",
        "expiresAt": "3022-01-24T19:37:37.030Z"
      },
      "datasetId": "ef99ab32-c2d4-4d1c-9e91-2c8be861bb8a",
      "datasetType": "VIRTUAL_DATASET",
      "datasetPath": [
        "Business",
        "Transportation",
        "NYC Taxi Trips"
      ],
      "canView": true,
      "canAlter": true,
      "entityType": "reflection-summary"
    },
    {
      "id": "51d76efd-225a-4799-a557-642810edef73",
      "type": "AGGREGATION",
      "name": "Aggregation Reflection",
      "createdAt": "2023-01-20T15:48:11.815Z",
      "updatedAt": "2023-01-20T15:48:11.815Z",
      "currentSizeBytes": 5488681,
      "totalSizeBytes": 16464596,
      "enabled": true,
      "arrowCachingEnabled": false,
      "status": {
        "config": "OK",
        "refresh": "SCHEDULED",
        "availability": "AVAILABLE",
        "combinedStatus": "CAN_ACCELERATE",
        "failureCount": 0,
        "lastDataFetch": "2023-01-25T16:35:20.689Z",
        "expiresAt": "2023-01-25T19:35:20.689Z"
      },
      "datasetId": "ed4b20c1-8ec5-4d4e-a2a0-f543bc74021b",
      "datasetType": "VIRTUAL_DATASET",
      "datasetPath": [
        "Business",
        "NYC-taxi-trips-by-distance"
      ],
      "canView": true,
      "canAlter": true,
      "entityType": "reflection-summary"
    },
    {
      "id": "fbd9f73a-d7a9-49d5-a2b6-efdd88fa9199",
      "type": "AGGREGATION",
      "name": "Aggregation Reflection",
      "createdAt": "2022-11-11T17:10:36.167Z",
      "updatedAt": "2022-11-11T17:10:36.167Z",
      "currentSizeBytes": 0,
      "totalSizeBytes": 0,
      "enabled": true,
      "arrowCachingEnabled": false,
      "status": {
        "config": "OK",
        "refresh": "GIVEN_UP",
        "availability": "NONE",
        "combinedStatus": "FAILED",
        "failureCount": 3,
        "lastDataFetch": "1969-12-31T23:59:59.999Z",
        "expiresAt": "1969-12-31T23:59:59.999Z"
      },
      "datasetId": "cfdc6530-8d5c-44d1-a85e-d43daca1c70f",
      "datasetType": "VIRTUAL_DATASET",
      "datasetPath": [
        "demo",
        "nyc-taxi-view-test"
      ],
      "canView": true,
      "canAlter": true,
      "entityType": "reflection-summary"
    },
    {
      "id": "14f22052-cbb3-4d5d-8bbc-6154cca98e49",
      "type": "RAW",
      "name": "listings",
      "createdAt": "2022-07-12T16:45:35.249Z",
      "updatedAt": "2022-07-12T16:45:35.249Z",
      "currentSizeBytes": 0,
      "totalSizeBytes": 0,
      "enabled": true,
      "arrowCachingEnabled": true,
      "status": {
        "config": "OK",
        "refresh": "MANUAL",
        "availability": "NONE",
        "combinedStatus": "CANNOT_ACCELERATE_MANUAL",
        "failureCount": 0,
        "lastDataFetch": "1969-12-31T23:59:59.999Z",
        "expiresAt": "1969-12-31T23:59:59.999Z"
      },
      "datasetId": "7707981c-cb33-42bc-a048-d27a8915f468",
      "datasetType": "PHYSICAL_DATASET",
      "datasetPath": [
        "Samples (9)",
        "samples.dremio.com",
        "Dremio University",
        "airbnb_listings.csv"
      ],
      "canView": true,
      "canAlter": true,
      "entityType": "reflection-summary"
    }
  ]
}

    Reflection Summary Attributes

    data

    [Object]

    List of reflection-summary objects for each reflection in the Dremio instance.

    data

    id

    String (UUID)

    Unique identifier of the reflection.

    Example 85dda9dd-2371-467f-b68d-fc4c5ea57a8b

    type

    String

    Reflection type. For more information, read Types of Reflections.

    Enum RAW , AGGREGATION

    Example AGGREGATION

    name

    String

    User-provided name for the reflection. For reflections created in the Dremio UI, if the user did not provide a name, the default values are Raw Reflection and Aggregation Reflection (automatically assigned based on the reflection type).

    Example Aggregation Reflection

    createdAt

    String

    Date and time that the reflection was created, in UTC format.

    Example 2022-07-05T19:19:40.244Z

    updatedAt

    String

    Date and time that the reflection was last updated, in UTC format.

    Example 2023-01-10T17:12:30.244Z

    currentSizeBytes

    Integer

    Data size of the latest reflection job (if one exists), in bytes.

    Example 18639885

    totalSizeBytes

    Integer

    Data size of all reflection jobs that have not been pruned (if any exist), in bytes.

    Example 142639924

    enabled

    Boolean

    If the reflection is available for accelerating queries, the value is true. Otherwise, the value is false.

    Example true

    arrowCachingEnabled

    Boolean

    If Dremio converts data from the reflection's Parquet files to Apache Arrow format when copying that data to executor nodes, the value is true. Otherwise, the value is false.

    Example false

    status

    Object

    Information about the status of the reflection.

    Example { "config": "OK", "refresh": "SCHEDULED", "availability": "AVAILABLE", "combinedStatus": "CAN_ACCELERATE", "failureCount": 0, "lastDataFetch": "2023-01-23T19:37:37.030Z", "expiresAt": "3022-01-24T19:37:37.030Z" }

    datasetId

    String

    Unique identifier of the anchor dataset that is associated with the reflection.

    Example ef99ab32-c2d4-4d1c-9e91-2c8be861bb8a

    datasetType

    String

    Type for the anchor dataset that is associated with the reflection. If the anchor dataset is a table, the type is PHYSICAL_DATASET. If the anchor dataset is a view, the type is VIRTUAL_DATASET.

    Enum PHYSICAL_DATASET , VIRTUAL_DATASET

    Example VIRTUAL_DATASET

    datasetPath

    [String]

    Path to the anchor dataset that is associated with the reflection within Dremio, expressed in an array. The path consists of the source or space, followed by any folder and subfolders, followed by the dataset itself as the last item in the array.

    Example [ "Business", "Transportation", "NYC Taxi Trips" ]

    canView

    Boolean

    If you can view reflections on all datasets of a source, system, space, or folder, the value is true. Otherwise, the value is false.

    Example true

    canAlter

    Boolean

    If you can create, edit, and view reflections on all datasets of a source, system, space, or folder, the value is true. Otherwise, the value is false.

    Example true

    entityType

    Boolean

    Type of the object. For reflection summaries, the entityType is reflection-summary.

    Example reflection-summary

    status

    config

    String

    Status of the reflection configuration. If the value is OK, the reflection configuration is free of errors. If the value is INVALID, the reflection configuration contains one or more errors.

    Enum OK , INVALID

    Example OK

    refresh

    String

    Status of the reflection refresh.

    GIVEN_UP: Dremio attempted to refresh the reflection multiple times, but each attempt has failed and Dremio will not make further attempts.
    MANUAL: Refresh period is set to 0, so you must use the Dremio UI to manually refresh the reflection.
    RUNNING: Dremio is currently refreshing the reflection.
    SCHEDULED: The reflection refreshes according to a schedule.

    Enum GIVEN_UP , MANUAL , RUNNING , SCHEDULED

    Example SCHEDULED

    availability

    String

    Status of the reflection's availability for accelerating queries.

    Enum NONE , INCOMPLETE , EXPIRED , AVAILABLE

    Example AVAILABLE

    combinedStatus

    String

    Status of the reflection based on a combination of config, refresh, and availability.

    CAN_ACCELERATE: The reflection is fully functional.
    CAN_ACCELERATE_WITH_FAILURES: The most recent refresh failed to obtain a status, but Dremio still has a valid materialization.
    CANNOT_ACCELERATE_MANUAL: The reflection is unable to accelerate any queries, and the Never Refresh option is selected for the refresh policy.
    CANNOT_ACCELERATE_SCHEDULED: The reflection is currently unable to accelerate any queries, but it has been scheduled for a refresh at a future time.
    DISABLED: The reflection has been manually disabled.
    EXPIRED: The reflection has expired and cannot be used.
    FAILED: The attempt to refresh the reflection has failed, typically three times in a row. The reflection is still usable.
    INVALID: The reflection is invalid because the underlying dataset has changed.
    INCOMPLETE: One or more pseudo-distributed file system (PDFS) nodes that contain materialized files are down (PFDS is supported for v21 and earlier). Only partial data is available. Configurations that use the Hadoop Distributed File System (HDFS) to store reflections should not experience incomplete status.
    REFRESHING: The reflection is currently being refreshed.

    Example CAN_ACCELERATE

    failureCount

    Integer

    Number of times that an attempt to refresh the reflection failed.

    Example 0

    lastDataFetch

    String

    Date and time that the reflection data was last refreshed, in UTC format. If the reflection is running, failing, or disabled, the lastDataFetch value is 1969-12-31T23:59:59.999Z.

    Example 2023-01-23T19:37:37.030Z

    expiresAt

    String

    Date and time that the reflection expires, in UTC format. If the reflection is running, failing, or disabled, the expiresAt value is 1969-12-31T23:59:59.999Z.

    Example 3022-01-24T19:37:37.030Z

    Retrieving a Reflection Summary

    Retrieve a summary of the raw and aggregation reflections in the Dremio instance.

    Method and URL 
    GET /api/v3/reflection/summary
    Example Request 
    curl -X GET 'https://{hostname}/v3/api/reflection/summary'
--header 'Authorization: _dremio{tokenstring}' \
--header "Content-Type: application/json"
    Example Response 
    {
  "data": [
    {
      "id": "85dda9dd-2371-467f-b68d-fc4c5ea57a8b",
      "type": "AGGREGATION",
      "name": "Aggregation Reflection",
      "createdAt": "2022-07-05T19:19:40.244Z",
      "updatedAt": "2023-01-10T17:12:30.244Z",
      "currentSizeBytes": 18639885,
      "totalSizeBytes": 142639924,
      "enabled": true,
      "arrowCachingEnabled": false,
      "status": {
        "config": "OK",
        "refresh": "SCHEDULED",
        "availability": "AVAILABLE",
        "combinedStatus": "CAN_ACCELERATE",
        "failureCount": 0,
        "lastDataFetch": "2023-01-23T19:37:37.030Z",
        "expiresAt": "3022-01-24T19:37:37.030Z"
      },
      "datasetId": "ef99ab32-c2d4-4d1c-9e91-2c8be861bb8a",
      "datasetType": "VIRTUAL_DATASET",
      "datasetPath": [
        "Business",
        "Transportation",
        "NYC Taxi Trips"
      ],
      "canView": true,
      "canAlter": true,
      "entityType": "reflection-summary"
    },
    {
      "id": "51d76efd-225a-4799-a557-642810edef73",
      "type": "AGGREGATION",
      "name": "Aggregation Reflection",
      "createdAt": "2023-01-20T15:48:11.815Z",
      "updatedAt": "2023-01-20T15:48:11.815Z",
      "currentSizeBytes": 5488681,
      "totalSizeBytes": 16464596,
      "enabled": true,
      "arrowCachingEnabled": false,
      "status": {
        "config": "OK",
        "refresh": "SCHEDULED",
        "availability": "AVAILABLE",
        "combinedStatus": "CAN_ACCELERATE",
        "failureCount": 0,
        "lastDataFetch": "2023-01-25T16:35:20.689Z",
        "expiresAt": "2023-01-25T19:35:20.689Z"
      },
      "datasetId": "ed4b20c1-8ec5-4d4e-a2a0-f543bc74021b",
      "datasetType": "VIRTUAL_DATASET",
      "datasetPath": [
        "Business",
        "NYC-taxi-trips-by-distance"
      ],
      "canView": true,
      "canAlter": true,
      "entityType": "reflection-summary"
    },
    {
      "id": "fbd9f73a-d7a9-49d5-a2b6-efdd88fa9199",
      "type": "AGGREGATION",
      "name": "Aggregation Reflection",
      "createdAt": "2022-11-11T17:10:36.167Z",
      "updatedAt": "2022-11-11T17:10:36.167Z",
      "currentSizeBytes": 0,
      "totalSizeBytes": 0,
      "enabled": true,
      "arrowCachingEnabled": false,
      "status": {
        "config": "OK",
        "refresh": "GIVEN_UP",
        "availability": "NONE",
        "combinedStatus": "FAILED",
        "failureCount": 3,
        "lastDataFetch": "1969-12-31T23:59:59.999Z",
        "expiresAt": "1969-12-31T23:59:59.999Z"
      },
      "datasetId": "cfdc6530-8d5c-44d1-a85e-d43daca1c70f",
      "datasetType": "VIRTUAL_DATASET",
      "datasetPath": [
        "demo",
        "nyc-taxi-view-test"
      ],
      "canView": true,
      "canAlter": true,
      "entityType": "reflection-summary"
    },
    {
      "id": "14f22052-cbb3-4d5d-8bbc-6154cca98e49",
      "type": "RAW",
      "name": "listings",
      "createdAt": "2022-07-12T16:45:35.249Z",
      "updatedAt": "2022-07-12T16:45:35.249Z",
      "currentSizeBytes": 0,
      "totalSizeBytes": 0,
      "enabled": true,
      "arrowCachingEnabled": true,
      "status": {
        "config": "OK",
        "refresh": "MANUAL",
        "availability": "NONE",
        "combinedStatus": "CANNOT_ACCELERATE_MANUAL",
        "failureCount": 0,
        "lastDataFetch": "1969-12-31T23:59:59.999Z",
        "expiresAt": "1969-12-31T23:59:59.999Z"
      },
      "datasetId": "7707981c-cb33-42bc-a048-d27a8915f468",
      "datasetType": "PHYSICAL_DATASET",
      "datasetPath": [
        "Samples (9)",
        "samples.dremio.com",
        "Dremio University",
        "airbnb_listings.csv"
      ],
      "canView": true,
      "canAlter": true,
      "entityType": "reflection-summary"
    }
  ]
}

    Response Status Codes

    200

    OK

    401

    Unauthorized

    404

    Not Found

    405

    Method Not Allowed