Reflection
Use the Reflection API to retrieve a list of raw and aggregation Reflections, retrieve individual Reflections, and create, update, and delete Reflections.
A Reflection is an optimized materialization of source data or a query, similar to a materialized view, that is derived from an existing table or view. The query optimizer can accelerate queries by using one or more Reflections to partially or entirely satisfy the queries rather than running queries against the raw data in the data source that underlies the table or view.
Reflection Object (Raw Reflection){
"id": "7a380a24-3b63-436c-9ea0-63cb534cc404",
"type": "RAW",
"name": "Raw Reflection",
"tag": "085e7704-544d-4c94-a666-2f298f5f8d7b",
"createdAt": "2023-01-30T14:11:43.826Z",
"updatedAt": "2023-01-30T14:11:43.826Z",
"datasetId": "tk973df7-ddf7-4d1e-fa9e-bccf28ae253f",
"currentSizeBytes": 4393709246,
"totalSizeBytes": 4393709246,
"enabled": true,
"status": {
"config": "OK",
"refresh": "SCHEDULED",
"availability": "AVAILABLE",
"combinedStatus": "CAN_ACCELERATE",
"failureCount": 0,
"lastDataFetch": "2023-01-30T14:11:51.801Z",
"expiresAt": "2023-01-30T17:11:51.801Z"
},
"displayFields": [
{
"name": "pickup_datetime"
},
{
"name": "passenger_count"
},
{
"name": "trip_distance_mi"
},
{
"name": "fare_amount"
},
{
"name": "tip_amount"
},
{
"name": "total_amount"
}
],
"distributionFields": [
{
"name": "trip_distance_mi"
}
],
"partitionFields": [
{
"name": "passenger_count"
}
],
"sortFields": [
{
"name": "pickup_datetime"
}
],
"partitionDistributionStrategy": "CONSOLIDATED",
"canView": true,
"canAlter": true,
"entityType": "reflection"
}
{
"id": "95dda9dd-2371-467f-b68d-fc4c5ea57a8b",
"type": "AGGREGATION",
"name": "Aggregation Reflection",
"tag": "05c6a6b5-1174-4f29-b1a3-b846ead498ce",
"createdAt": "2022-07-05T19:19:40.244Z",
"updatedAt": "2023-01-10T17:12:40.244Z",
"datasetId": "df99ab32-c2d4-4d1c-9e91-2c8be861bb8a",
"currentSizeBytes": 18639885,
"totalSizeBytes": 142639924,
"enabled": true,
"status": {
"config": "OK",
"refresh": "SCHEDULED",
"availability": "AVAILABLE",
"combinedStatus": "CAN_ACCELERATE",
"failureCount": 0,
"lastDataFetch": "2023-01-10T17:12:40.244Z",
"expiresAt": "3022-07-05T19:19:40.244Z"
},
"dimensionFields": [
{
"name": "pickup_date",
"granularity": "DATE"
},
{
"name": "pickup_datetime",
"granularity": "DATE"
},
{
"name": "dropoff_date",
"granularity": "DATE"
},
{
"name": "dropoff_datetime",
"granularity": "DATE"
},
{
"name": "passenger_count",
"granularity": "DATE"
},
{
"name": "total_amount",
"granularity": "DATE"
},
{
"name": "trip_distance_mi",
"granularity": "DATE"
}
],
"measureFields": [
{
"name": "passenger_count",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "trip_distance_mi",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "fare_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "surcharge",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "tip_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "total_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
}
],
"distributionFields": [
{
"name": "trip_distance_mi"
},
{
"name": "total_amount"
}
],
"partitionFields": [
{
"name": "dropoff_date"
},
{
"name": "passenger_count"
}
],
"sortFields": [
{
"name": "trip_distance_mi"
}
],
"partitionDistributionStrategy": "CONSOLIDATED",
"canView": true,
"canAlter": true,
"entityType": "reflection"
}
Reflection Attributes
id String (UUID)
UUID for the Reflection.
type String
Reflection type. For details, read Types of Reflections.
Valid Values: RAW, AGGREGATION
Example: AGGREGATION
name String
User-provided name for the Reflection. For Reflections created in the Dremio console, if the user did not provide a name, the default values are Raw Reflection and Aggregation Reflection (automatically assigned based on the Reflection type).
Example: Aggregation Reflection
tag String (UUID)
UUID for the version of the Reflection. Dremio changes the tag whenever the Reflection changes and uses the tag value to ensure that PUT requests apply to the most recent version of the Reflection.
createdAt String
Timestamp when the Reflection was created.
updatedAt String
Timestamp when the Reflection was updated.
datasetId String (UUID)
UUID of the dataset.
currentSizeBytes Integer
Data size of the latest Reflection job (if one exists). In bytes.
Example: 18639885
totalSizeBytes Integer
Data size of all Reflection jobs that have not been pruned (if any exist). In bytes.
Example: 142639924
enabled Boolean
If the Reflection is available for accelerating queries, true. Otherwise, false.
status Object
Information about the status of the Reflection.
Example:
{
"config": "OK",
"refresh": "SCHEDULED",
"availability": "AVAILABLE",
"combinedStatus": "CAN_ACCELERATE",
"failureCount": 0,
"lastDataFetch": "2023-01-10T17:12:40.244Z",
"expiresAt": "3022-07-05T19:19:40.244Z"
}
displayFields Array of Object
Information about the fields displayed from the anchor dataset. Each displayFields object contains one attribute: name. Valid only for raw Reflections.
Example:
[
{
"name": "pickup_datetime"
},
{
"name": "passenger_count"
},
{
"name": "trip_distance_mi"
},
{
"name": "fare_amount"
},
{
"name": "tip_amount"
},
{
"name": "total_amount"
}
]
dimensionFields Array of Object
Information about the dimension fields from the anchor dataset used in the Reflection. Dimension fields are the fields you expect to group by when analyzing data. Each dimensionFields object contains two attributes: name and granularity. Valid only for aggregation Reflections.
Example:
[
{
"name": "pickup_date",
"granularity": "DATE"
},
{
"name": "pickup_datetime",
"granularity": "DATE"
},
{
"name": "dropoff_date",
"granularity": "DATE"
},
{
"name": "dropoff_datetime",
"granularity": "DATE"
},
{
"name": "passenger_count",
"granularity": "DATE"
},
{
"name": "total_amount",
"granularity": "DATE"
}
]
measureFields Array of Object
Information about the measure fields from the anchor dataset used in the Reflection. Measure fields are the fields you expect to use for calculations when analyzing the data. Each measureFields object contains two attributes: name and measureTypeList. Valid only for aggregation Reflections.
Example:
[
{
"name": "passenger_count",
"measureTypeList": [
"SUM,
"COUNT"
]
},
{
"name": "trip_distance_mi",
"measureTypeList": [
"SUM",
"COUNT"
]
}
]
distributionFields Array of Object
Information about the distribution fields from the anchor dataset used in the Reflection. Distribution fields allow data from multiple datasets to be co-located and co-partitioned across nodes to minimize data movement during join operations. Each distributionFields object contains one attribute: name.
Example:
[
{
"name": "trip_distance_mi"
},
{
"name": "total_amount"
}
]
partitionFields Array of Object
Information about the fields from the anchor dataset used to partition data in the Reflection. Each field name is listed as an individual object. For details, read Horizontally Partition Reflections that Have Many Rows.
Example:
[
{
"name": "dropoff_date"
},
{
"name": "passenger_count"
}
]
sortFields Array of Object
Information about the fields from the anchor dataset used for sorting in the Reflection. Each sortFields object contains one attribute: name. For details, read Sort Reflections on High-Cardinality Fields.
Example:
[
{
"name": "trip_distance_mi"
}
]
partitionDistributionStrategy String
Method used to optimize data compression when executing Reflections. CONSOLIDATED means Dremio minimizes the number of files produced. The query threads pool the data and ensure that the fewest number of files are written to the Reflection store. Optimizing for a smaller number of files generally improves read performance because users can perform fewer searches for a given query. STRIPED means Dremio minimizes the time required to refresh the Reflection. Each final-stage query thread opens its own writers to write the data, which can result in many small files if each query thread contains a small amount of data.
Valid Values: CONSOLIDATED, STRIPED
Example: CONSOLIDATED
canView Boolean
If you can view Reflections on all datasets of a project, source, or folder, true. Otherwise, false.
canAlter Boolean
If you can create, edit, and view Reflections on all datasets of a source, system, or folder, true. Otherwise, false.
entityType String
Type of entity. For Reflection objects, the entityType is reflection.
Example: reflection
Attributes of the status Object
config String
Status of the Reflection configuration. If OK, the Reflection configuration is free of errors. If INVALID, the Reflection configuration contains one or more errors.
Valid Values: OK, INVALID
Example: OK
refresh String
Status of the Reflection refresh.
GIVEN_UP: Dremio attempted to refresh the Reflection multiple times, but each attempt has failed and Dremio will not make further attempts.MANUAL: Refresh period is set to 0, so you must use the Dremio console to manually refresh the Reflection.RUNNING: Dremio is currently refreshing the Reflection.SCHEDULED: The Reflection refreshes according to a schedule.
Valid Values: GIVEN_UP, MANUAL, RUNNING, SCHEDULED
Example: SCHEDULED
availability String
Status of the Reflection's availability for accelerating queries.
Valid Values: NONE, EXPIRED, AVAILABLE
Example: AVAILABLE
combinedStatus String
Status of the Reflection based on a combination of config, refresh, and availability statuses.
CAN_ACCELERATE: The Reflection is fully functional.CAN_ACCELERATE_WITH_FAILURES: The most recent refresh failed to obtain a status, but Dremio still has a valid materialization.CANNOT_ACCELERATE_MANUAL: The Reflection is unable to accelerate any queries, and theNever Refreshoption is selected for the refresh policy.CANNOT_ACCELERATE_SCHEDULED: The Reflection is currently unable to accelerate any queries, but it has been scheduled for a refresh at a future time.DISABLED: The Reflection has been manually disabled.EXPIRED: The Reflection has expired and cannot be used.FAILED: The attempt to refresh the Reflection has failed, typically three times in a row. The Reflection is still usable.INVALID: The Reflection is invalid because the underlying dataset has changed.REFRESHING: The Reflection is currently being refreshed.
Example: CAN_ACCELERATE
failureCount Integer
Number of times that an attempt to refresh the Reflection failed.
lastDataFetch String
Timestamp when the Reflection was last refreshed.
expiresAt String
Timestamp when the Reflection will expire.
Attributes of Objects in the displayFields Array
name String
Name of the field from the anchor dataset that is displayed in the raw Reflection.
Example: passenger_count
Attributes of Objects in the dimensionFields Array
name String
Name of the field from the anchor dataset that is configured as a dimension for the Reflection.
Example: pickup_date
granularity String
Grouping used for the dimension field. When timestamp and date fields are configured as dimensions, Dremio can automatically extract and use the day-level date value (DATE) or use the field's original value (NORMAL).
Valid Values: DATE, NORMAL
Example: DATE
Attributes of Objects in the measureFields Array
name String
Name of the field from the anchor dataset that is configured as a measure for the Reflection.
Example: passenger_count
measureTypeList Array of String
Types of calculations for which Dremio uses the specified measure field.
Valid Values: APPROX_COUNT_DISTINCT, MIN, MAX, UNKNOWN, SUM, COUNT
Example:
[
"SUM",
"COUNT"
]
Attributes of Objects in the distributionFields Array
name String
Name of the field from the anchor dataset that is used for co-locating and co-partitioning data from multiple datasets across nodes.
Example: trip_distance_mi
Attributes of Objects in the partitionFields Array
name String
Name of the field from the anchor dataset that is used for sorting in the Reflection.
Example: dropoff_date
Attributes of Objects in the sortFields Array
name String
Name of the field from the anchor dataset on which you can partition the rows in the Reflection.
Example: trip_distance_mi
Create a Reflection
Method and URLPOST /v0/projects/{project_id}/reflection
Parameters
project_id Path String (UUID)
type Body String
Reflection type. For details, read Types of Reflections.
Valid Values: RAW, AGGREGATION
Example: AGGREGATION
name Body String
Name to use for the Reflection.
Example: New Aggregation Reflection
datasetId Body String (UUID)
UUID of the dataset.
enabled Body Boolean
If the Reflection should be available for accelerating queries, true. Otherwise, false.
displayFields Body Array of Object Optional
Information about the fields to display from the anchor dataset. The displayfields array must list every field in the anchor dataset or the Reflection will fail. Each displayFields object contains one attribute: name. Valid only for raw Reflections.
Example:
[
{
"name": "pickup_datetime"
},
{
"name": "passenger_count"
},
{
"name": "trip_distance_mi"
},
{
"name": "fare_amount"
},
{
"name": "tip_amount"
},
{
"name": "total_amount"
}
]
dimensionFields Body Array of Object Optional
Information about the dimension fields from the anchor dataset to use in the Reflection. Dimension fields are the fields you expect to group by when analyzing data. Each dimensionFields object contains two attributes: name and granularity. Valid only for aggregation Reflections.
Example:
[
{
"name": "pickup_date",
"granularity": "DATE"
},
{
"name": "pickup_datetime",
"granularity": "DATE"
},
{
"name": "dropoff_date",
"granularity": "DATE"
},
{
"name": "dropoff_datetime",
"granularity": "DATE"
},
{
"name": "passenger_count",
"granularity": "DATE"
},
{
"name": "total_amount",
"granularity": "DATE"
}
]
measureFields Body Array of Object Optional
Information about the measure fields from the anchor dataset to use in the Reflection. Measure fields are the fields you expect to use for calculations when analyzing the data. Each measureFields object contains two attributes: name and measureTypeList. Valid only for aggregation Reflections.
Example: [\{"name": "passenger_count","measureTypeList": ["SUM,"COUNT"]},\{"name": "trip_distance_mi","measureTypeList": ["SUM","COUNT"]},\{"name": "fare_amount","measureTypeList": ["SUM","COUNT"]},\{"name": "surcharge","measureTypeList": ["SUM","COUNT"]},\{"name": "tip_amount","measureTypeList": ["SUM","COUNT"]},\{"name": "total_amount","measureTypeList": ["SUM","COUNT"]}]
distributionFields Body Array of Object Optional
Information about the distribution fields from the anchor dataset to use for co-locating and co-partitioning data from multiple datasets across nodes. Each distributionFields object contains one attribute: name.
Example:
[
{
"name": "trip_distance_mi"
},
{
"name": "total_amount"
}
]
partitionFields Body Array of Object Optional
Information about the fields from the anchor dataset to use to partition data in the Reflection. Each field name is listed as an individual object. For details, read Horizontally Partition Reflections that Have Many Rows.
Example:
[
{
"name": "dropoff_date"
},
{
"name": "passenger_count"
}
]
sortFields Body Array of Object Optional
Information about the fields from the anchor dataset to use for sorting in the Reflection. Each sortFields object contains one attribute: name. For details, read Sort Reflections on High-Cardinality Fields.
Example:
[
{
"name": "trip_distance_mi"
}
]
partitionDistributionStrategy Body String Optional
Method to use to optimize data compression when executing Reflections. If CONSOLIDATED (default), Dremio will minimize the number of files produced. If STRIPED, Dremio will minimize the time required to refresh the Reflection.
Valid Values: CONSOLIDATED, STRIPED
Example: CONSOLIDATED
canView Body Boolean Optional
To allow users to view Reflections on all datasets of a project, source, or folder, true (default). Otherwise, false.
canAlter Body Boolean Optional
To allow users to create, edit, and view Reflections on all datasets of a source, system, or folder, true (default). Otherwise, false.
entityType Body String
Type of entity. For Reflection objects, the entityType is reflection.
Parameters of Objects in the displayFields Array
name Body String
Name of the field to display from the anchor dataset.
Example: pickup_datetime
Parameters of Objects in the dimensionFields Array
name Body String
Name of the field from the anchor dataset to configure as a dimension for the Reflection.
Example: pickup_datetime
granularity Body String
Grouping to use for the dimension field. If Dremio should automatically extract the day-level date value and use it as the grouping value in the Reflection, DATE. If Dremio should use the original value for grouping, NORMAL.
Valid Values: DATE, NORMAL
Example: DATE
Parameters of Objects in the measureFields Array
name Body String
Name of the field from the anchor dataset that you expect to use in calculations. Fields of types LIST, MAP, and UNION are not valid measureFields.
Example: passenger_count
measureTypeList Body Array of String
Types of calculations for which Dremio should use the specified measure field. The calculations must be valid for the specified field (for example, SUM is not valid for a timestamp field like pickup_datetime).
Valid Values: APPROX_COUNT_DISTINCT, MIN, MAX, UNKNOWN, SUM, COUNT
Example:
[
"SUM",
"COUNT"
]
Parameters of Objects in the distributionFields Array
name Body String Optional
Name of the field from the anchor dataset to use for co-locating and co-partitioning data from multiple datasets across nodes. In aggregation Reflections, every field listed as a distribution field must also be listed as a dimension field.
Example: trip_distance_mi
Parameters of Objects in the partitionFields Array
name Body String Optional
Name of the field from the anchor dataset on which you want to be able to partition rows. Every field listed as a partition field must also be listed as a dimension field. If you list a field as a partition field, you cannot list the same field as a sort field in the same Reflection.
Example: pickup_datetime
Parameters of Objects in the sortFields Array
name Body String Optional
Name of the field from the anchor dataset to use for sorting in the Reflection. Every field listed as a sort field must also be listed as a dimension field. If you list a field as a sort field, you cannot list the same field as a partition field in the same Reflection.
Example: trip_distance_mi
Example
Requestcurl -X POST "https://api.dremio.cloud/v0/projects/$PROJECT_ID/reflection/" \
-H "Authorization: Bearer $DREMIO_TOKEN" \
-H 'Content-Type: application/json' \
--data-raw '{
"type": "AGGREGATION",
"name": "New Aggregation Reflection",
"datasetId": "81e2ad31-a119-447d-a831-085831e505be",
"enabled": true,
"dimensionFields": [
{
"name": "pickup_datetime",
"granularity": "DATE"
},
{
"name": "passenger_count",
"granularity": "DATE"
},
{
"name": "total_amount",
"granularity": "DATE"
},
{
"name": "trip_distance_mi",
"granularity": "DATE"
}
],
"measureFields": [
{
"name": "passenger_count",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "trip_distance_mi",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "fare_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "tip_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "total_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
}
],
"distributionFields": [
{
"name": "trip_distance_mi"
},
{
"name": "total_amount"
}
],
"partitionFields": [
{
"name": "pickup_datetime"
},
{
"name": "passenger_count"
}
],
"sortFields": [
{
"name": "trip_distance_mi"
}
],
"entityType": "reflection"
}'
{
"id": "26526a2e-3edb-4af0-a24f-8e9e2dcfa82a",
"type": "AGGREGATION",
"name": "New Aggregation Reflection",
"tag": "ff9f4477-e44d-4995-a76a-70edbe572c56",
"createdAt": "2023-01-30T14:30:24.311Z",
"updatedAt": "2023-01-30T14:30:24.311Z",
"datasetId": "81e2ad31-a119-447d-a831-085831e505be",
"currentSizeBytes": 0,
"totalSizeBytes": 0,
"enabled": true,
"status": {
"config": "OK",
"refresh": "SCHEDULED",
"availability": "NONE",
"combinedStatus": "CANNOT_ACCELERATE_SCHEDULED",
"failureCount": 0,
"lastDataFetch": "1969-12-31T23:59:59.999Z",
"expiresAt": "1969-12-31T23:59:59.999Z"
},
"dimensionFields": [
{
"name": "pickup_datetime",
"granularity": "DATE"
},
{
"name": "passenger_count",
"granularity": "DATE"
},
{
"name": "total_amount",
"granularity": "DATE"
},
{
"name": "trip_distance_mi",
"granularity": "DATE"
}
],
"measureFields": [
{
"name": "passenger_count",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "trip_distance_mi",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "fare_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "tip_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "total_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
}
],
"distributionFields": [
{
"name": "trip_distance_mi"
},
{
"name": "total_amount"
}
],
"partitionFields": [
{
"name": "pickup_datetime"
},
{
"name": "passenger_count"
}
],
"sortFields": [
{
"name": "trip_distance_mi"
}
],
"partitionDistributionStrategy": "CONSOLIDATED",
"canView": true,
"canAlter": true,
"entityType": "reflection"
}
Response Status Codes
200 OK
400 Bad Request
401 Unauthorized
405 Method Not Allowed
500 Internal Server Error
Retrieve All Reflections
Method and URLGET /v0/projects/{project_id}/reflection
Parameters
project_id Path String (UUID)
Example
Requestcurl -X GET "https://api.dremio.cloud/v0/projects/$PROJECT_ID/reflection/" \
-H "Authorization: Bearer $DREMIO_TOKEN" \
-H 'Content-Type: application/json'
In the response for a request to retrieve all raw and aggregation Reflections, the Reflection objects are wrapped with a data array. Each object in the data array represents one Reflection.
Response{
"data": [
{
"id": "95dda9dd-2371-467f-b68d-fc4c5ea57a8b",
"type": "AGGREGATION",
"name": "Aggregation Reflection",
"tag": "05c6a6b5-1174-4f29-b1a3-b846ead498ce",
"createdAt": "2022-07-05T19:19:40.244Z",
"updatedAt": "2023-01-10T17:12:40.244Z",
"datasetId": "df99ab32-c2d4-4d1c-9e91-2c8be861bb8a",
"currentSizeBytes": 18639885,
"totalSizeBytes": 142639924,
"enabled": true,
"status": {
"config": "OK",
"refresh": "SCHEDULED",
"availability": "AVAILABLE",
"combinedStatus": "CAN_ACCELERATE",
"failureCount": 0,
"lastDataFetch": "2023-01-10T17:12:40.244Z",
"expiresAt": "3022-07-05T19:19:40.244Z"
},
"dimensionFields": [
{
"name": "pickup_date",
"granularity": "DATE"
},
{
"name": "pickup_datetime",
"granularity": "DATE"
},
{
"name": "dropoff_date",
"granularity": "DATE"
},
{
"name": "dropoff_datetime",
"granularity": "DATE"
},
{
"name": "passenger_count",
"granularity": "DATE"
},
{
"name": "total_amount",
"granularity": "DATE"
},
{
"name": "trip_distance_mi",
"granularity": "DATE"
}
],
"measureFields": [
{
"name": "passenger_count",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "trip_distance_mi",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "fare_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "surcharge",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "tip_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "total_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
}
],
"distributionFields": [
{
"name": "trip_distance_mi"
},
{
"name": "total_amount"
}
],
"partitionFields": [
{
"name": "dropoff_date"
},
{
"name": "passenger_count"
}
],
"sortFields": [
{
"name": "trip_distance_mi"
}
],
"partitionDistributionStrategy": "CONSOLIDATED",
"canView": true,
"canAlter": true,
"entityType": "reflection"
},
{
"id": "14f22052-cbb3-4d5d-8bbc-6154cca98e49",
"type": "RAW",
"name": "listings",
"tag": "7707981c-c2d4-4d1c-9e91-2c8be861bb8a",
"createdAt": "2022-07-12T16:45:35.249Z",
"updatedAt": "2022-07-12T16:45:35.249Z",
"datasetId": "df99ab32-cb33-42bc-a048-d27a8915f468",
"currentSizeBytes": 0,
"totalSizeBytes": 0,
"enabled": true,
"status": {
"config": "OK",
"refresh": "MANUAL",
"availability": "NONE",
"combinedStatus": "CANNOT_ACCELERATE_MANUAL",
"failureCount": 0,
"lastDataFetch": "1969-12-31T23:59:59.999Z",
"expiresAt": "1969-12-31T23:59:59.999Z"
},
"displayFields": [
{
"name": "id"
}
],
"partitionDistributionStrategy": "CONSOLIDATED",
"canView": true,
"canAlter": true,
"entityType": "reflection"
},
{
"id": "6c209200-b522-4f81-bbe0-d10668c7752c",
"type": "AGGREGATION",
"name": "Aggregation Reflection",
"tag": "bbd9a011-3062-489d-b1ca-d775947b4bbe",
"createdAt": "2021-09-29T15:47:44.806Z",
"updatedAt": "2021-09-29T15:47:44.806Z",
"datasetId": "746f867a-c27c-4711-bb8c-99546a4c25e0",
"currentSizeBytes": 0,
"totalSizeBytes": 1675978,
"enabled": true,
"status": {
"config": "OK",
"refresh": "GIVEN_UP",
"availability": "NONE",
"combinedStatus": "FAILED",
"failureCount": 3,
"lastDataFetch": "1969-12-31T23:59:59.999Z",
"expiresAt": "1969-12-31T23:59:59.999Z"
},
"dimensionFields": [
{
"name": "passenger_count",
"granularity": "DATE"
},
{
"name": "pickup_datetime",
"granularity": "DATE"
}
],
"measureFields": [
{
"name": "trip_distance_mi",
"measureTypeList": [
"COUNT",
"SUM"
]
},
{
"name": "total_amount",
"measureTypeList": [
"COUNT",
"SUM"
]
},
{
"name": "tip_amount",
"measureTypeList": [
"COUNT",
"SUM"
]
},
{
"name": "fare_amount",
"measureTypeList": [
"COUNT",
"SUM"
]
}
],
"partitionDistributionStrategy": "CONSOLIDATED",
"canView": true,
"canAlter": true,
"entityType": "reflection"
},
{
"id": "c5c5b282-ffea-4a34-835f-cc591584412b",
"type": "AGGREGATION",
"name": "Test reflection",
"tag": "e6dd9a96-a069-4623-a672-75f9d9355bfd",
"createdAt": "2021-10-11T18:44:27.064Z",
"updatedAt": "2021-10-11T18:44:27.064Z",
"datasetId": "316531b8-3c56-42f2-b05f-81f228ef3162",
"currentSizeBytes": 0,
"totalSizeBytes": 0,
"enabled": true,
"status": {
"config": "OK",
"refresh": "MANUAL",
"availability": "NONE",
"combinedStatus": "CANNOT_ACCELERATE_MANUAL",
"failureCount": 0,
"lastDataFetch": "1969-12-31T23:59:59.999Z",
"expiresAt": "1969-12-31T23:59:59.999Z"
},
"dimensionFields": [
{
"name": "passenger_count",
"granularity": "DATE"
}
],
"measureFields": [
{
"name": "trip_distance_mi",
"measureTypeList": [
"COUNT",
"SUM"
]
},
{
"name": "total_amount",
"measureTypeList": [
"COUNT",
"SUM"
]
},
{
"name": "tip_amount",
"measureTypeList": [
"COUNT",
"SUM"
]
},
{
"name": "fare_amount",
"measureTypeList": [
"COUNT",
"SUM"
]
}
],
"partitionDistributionStrategy": "CONSOLIDATED",
"canView": true,
"canAlter": true,
"entityType": "reflection"
}
],
"canAlterReflections": true
}
Response Status Codes
200 OK
401 Unauthorized
404 Not Found
500 Internal Server Error
Retrieve a Reflection
Method and URLGET /v0/projects/{project_id}/reflection/{id}
Parameters
project_id Path String (UUID)
id Path String (UUID)
UUID for the Reflection that you want to retrieve.
Example
Requestcurl -X GET "https://api.dremio.cloud/v0/projects/$PROJECT_ID/reflection/$REFLECTION_ID"
-H 'Authorization: Bearer <access_token>' \
-H 'Content-Type: application/json'
{
"id": "95dda9dd-2371-467f-b68d-fc4c5ea57a8b",
"type": "AGGREGATION",
"name": "Aggregation Reflection",
"tag": "05c6a6b5-1174-4f29-b1a3-b846ead498ce",
"createdAt": "2022-07-05T19:19:40.244Z",
"updatedAt": "2023-01-10T17:12:40.244Z",
"datasetId": "df99ab32-c2d4-4d1c-9e91-2c8be861bb8a",
"currentSizeBytes": 18639885,
"totalSizeBytes": 142639924,
"enabled": true,
"status": {
"config": "OK",
"refresh": "SCHEDULED",
"availability": "AVAILABLE",
"combinedStatus": "CAN_ACCELERATE",
"failureCount": 0,
"lastDataFetch": "2023-01-10T17:12:40.244Z",
"expiresAt": "3022-07-05T19:19:40.244Z"
},
"dimensionFields": [
{
"name": "pickup_date",
"granularity": "DATE"
},
{
"name": "pickup_datetime",
"granularity": "DATE"
},
{
"name": "dropoff_date",
"granularity": "DATE"
},
{
"name": "dropoff_datetime",
"granularity": "DATE"
},
{
"name": "passenger_count",
"granularity": "DATE"
},
{
"name": "total_amount",
"granularity": "DATE"
},
{
"name": "trip_distance_mi",
"granularity": "DATE"
}
],
"measureFields": [
{
"name": "passenger_count",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "trip_distance_mi",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "fare_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "surcharge",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "tip_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "total_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
}
],
"distributionFields": [
{
"name": "trip_distance_mi"
},
{
"name": "total_amount"
}
],
"partitionFields": [
{
"name": "dropoff_date"
},
{
"name": "passenger_count"
}
],
"sortFields": [
{
"name": "trip_distance_mi"
}
],
"partitionDistributionStrategy": "CONSOLIDATED",
"canView": true,
"canAlter": true,
"entityType": "reflection"
}
Response Status Codes
200 OK
401 Unauthorized
404 Not Found
500 Internal Server Error
Retrieve All Reflections for a Dataset
Method and URLGET /v0/projects/{project_id}/dataset/{datasetId}/reflection
Parameters
project_id Path String (UUID)
datasetId Path String (UUID)
UUID of the dataset.
Example
Requestcurl -X GET "https://api.dremio.cloud/v0/projects/$PROJECT_ID/dataset/$DATASET_ID/reflection"
-H 'Authorization: Bearer <access_token>' \
-H 'Content-Type: application/json'
In the response for a request to retrieve all raw and aggregation Reflections for a dataset, the Reflection objects are wrapped with a data array. Each object in the data array represents one Reflection.
Response{
"data": [
{
"id": "23f75eb1-045f-447f-b3fa-374377877569",
"type": "RAW",
"name": "Raw Reflection",
"tag": "41deaef2-58a1-40b3-9ff4-6418b881d881",
"createdAt": "2023-02-03T16:38:27.770Z",
"updatedAt": "2023-02-03T16:38:27.770Z",
"datasetId": "3cbab7b3-ee82-44c1-abcc-e86d56078d4d",
"currentSizeBytes": 43286,
"totalSizeBytes": 87522,
"enabled": true,
"status": {
"config": "OK",
"refresh": "MANUAL",
"availability": "AVAILABLE",
"combinedStatus": "CAN_ACCELERATE",
"failureCount": 0,
"lastDataFetch": "2023-02-03T16:38:27.780Z",
"expiresAt": "3022-06-06T16:38:27.780Z"
},
"displayFields": [
{
"name": "pickup_datetime"
},
{
"name": "passenger_count"
},
{
"name": "trip_distance_mi"
},
{
"name": "fare_amount"
},
{
"name": "tip_amount"
},
{
"name": "total_amount"
}
],
"partitionDistributionStrategy": "CONSOLIDATED",
"canView": true,
"canAlter": true,
"entityType": "reflection"
},
{
"id": "4f7c303d-fa6e-4f80-8c6a-cd8243095de9",
"type": "AGGREGATION",
"name": "Aggregation Reflection",
"tag": "bc5fa8f9-4af5-47de-b234-66f2b401c90e",
"createdAt": "2023-02-03T16:39:40.556Z",
"updatedAt": "2023-02-03T16:39:40.556Z",
"datasetId": "1acab7b3-ee82-44c1-abcc-e86d56078d4d",
"currentSizeBytes": 31579,
"totalSizeBytes": 124983,
"enabled": true,
"status": {
"config": "OK",
"refresh": "MANUAL",
"availability": "AVAILABLE",
"combinedStatus": "CAN_ACCELERATE",
"failureCount": 0,
"lastDataFetch": "2023-02-03T16:39:40.568Z",
"expiresAt": "3022-06-06T16:39:40.568Z"
},
"dimensionFields": [
{
"name": "passenger_count",
"granularity": "DATE"
},
{
"name": "pickup_datetime",
"granularity": "DATE"
}
],
"measureFields": [
{
"name": "trip_distance_mi",
"measureTypeList": [
"COUNT",
"SUM"
]
},
{
"name": "total_amount",
"measureTypeList": [
"COUNT",
"SUM"
]
},
{
"name": "tip_amount",
"measureTypeList": [
"COUNT",
"SUM"
]
},
{
"name": "fare_amount",
"measureTypeList": [
"COUNT",
"SUM"
]
}
],
"partitionDistributionStrategy": "CONSOLIDATED",
"canView": true,
"canAlter": true,
"entityType": "reflection"
}
],
"canAlterReflections": true
}
Response Status Codes
200 OK
401 Unauthorized
404 Not Found
405 Method Not Allowed
500 Internal Server Error
Refresh a Reflection
For information about the refresh action performed, see Trigger Reflection Refreshes
Method and URLPOST /v0/projects/{project_id}/reflection/{id}/refresh
Parameters
project_id Path String (UUID)
id Path String (UUID)
UUID for the Reflection that you want to base the refresh action on.
Response Status Codes
200 OK
400 Not supported
401 Unauthorized
404 Not Found
405 Method Not Allowed
500 Internal Server Error
Update a Reflection
Method and URLPUT /v0/projects/{project_id}/reflection/{id}
Parameters
project_id Path String (UUID)
id Path String (UUID)
UUID for the Reflection that you want to update.
type Body String
Reflection type. For details, read Types of Reflections.
Valid Values: RAW, AGGREGATION
Example: AGGREGATION
name Body String
Name to use for the Reflection.
Example: New Aggregation Reflection
tag Body String (UUID)
UUID of the version of the Reflection. Dremio uses the tag value to ensure that you are updating the most recent version of the Reflection.
datasetId Body String (UUID)
UUID of the dataset.
enabled Body Boolean
If the Reflection should be available for accelerating queries, true. Otherwise, false.
displayFields Body Array of Object Optional
Information about the fields to display from the anchor dataset. The displayfields array must list every field in the anchor dataset or the Reflection will fail. Each displayFields object contains one attribute: name. Valid only for raw Reflections.
Example:
[
{
"name": "pickup_datetime"
},
{
"name": "passenger_count"
},
{
"name": "trip_distance_mi"
},
{
"name": "fare_amount"
},
{
"name": "tip_amount"
},
{
"name": "total_amount"
}
]
dimensionFields Body Array of Object Optional
Information about the dimension fields from the anchor dataset to use in the Reflection. Dimension fields are the fields you expect to group by when analyzing data. Each dimensionFields object contains two attributes: name and granularity. Valid only for aggregation Reflections. If you omit the dimensionFields object in a PUT request, Dremio will remove all existing dimension fields from the Reflection. To keep existing dimension fields while making other updates, duplicate the existing dimensionFields array in the PUT request.
Example:
[
{
"name": "pickup_datetime",
"granularity": "DATE"
},
{
"name": "passenger_count",
"granularity": "DATE"
},
{
"name": "total_amount",
"granularity": "DATE"
},
{
"name": "trip_distance_mi",
"granularity": "DATE"
}
]
measureFields Body Array of Object Optional
Information about the measure fields from the anchor dataset to use in the Reflection. Measure fields are the fields you expect to use for calculations when analyzing the data. Each measureFields object contains two attributes: name and measureTypeList. Valid only for aggregation Reflections. If you omit the measureFields object in a PUT request, Dremio will remove all existing measure fields from the Reflection. To keep existing measure fields while making other updates, duplicate the existing measureFields array in the PUT request.
Example:
[
{
"name": "passenger_count",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "trip_distance_mi",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "fare_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "tip_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "total_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
}
]
distributionFields Body Array of Object Optional
Information about the distribution fields from the anchor dataset to use for co-locating and co-partitioning data from multiple datasets across nodes. Each distributionFields object contains one attribute: name. If you omit the distributionFields object in a PUT request, Dremio will remove all existing distribution fields from the Reflection. To keep existing distribution fields while making other updates, duplicate the existing distributionFields array in the PUT request.
Example:
[
{
"name": "trip_distance_mi"
},
{
"name": "total_amount"
}
]
partitionFields Body Array of Object Optional
Information about the fields from the anchor dataset to use to partition data in the Reflection. Each field name is listed as an individual object. For details, read Horizontally Partition Reflections that Have Many Rows. If you omit the partitionFields object in a PUT request, Dremio will remove all existing partition fields from the Reflection. To keep existing partition fields while making other updates, duplicate the existing partitionFields array in the PUT request.
Example:
[
{
"name": "pickup_datetime"
},
{
"name": "passenger_count"
}
]
sortFields Body Array of Object Optional
Information about the fields from the anchor dataset to use for sorting in the Reflection. Each sortFields object contains one attribute: name. For details, read Sort Reflections on High-Cardinality Fields. If you omit the sortFields object in a PUT request, Dremio will remove all existing sort fields from the Reflection. To keep existing sort fields while making other updates, duplicate the existing sortFields array in the PUT request.
Example:
[
{
"name": "trip_distance_mi"
}
]
partitionDistributionStrategy Body String Optional
Method to use to optimize data compression when executing Reflections. If CONSOLIDATED (default), Dremio will minimize the number of files produced. If STRIPED, Dremio will minimize the time required to refresh the Reflection.
Valid Values: CONSOLIDATED, STRIPED
Example: CONSOLIDATED
Parameters of Objects in the displayFields Array
name Body String
Name of the field to display from the anchor dataset.
Example: pickup_datetime
Parameters of Objects in the dimensionFields Array
name Body String
Name of the field from the anchor dataset to configure as a dimension for the Reflection.
Example: pickup_datetime
granularity Body String
Grouping to use for the dimension field. If Dremio should automatically extract the day-level date value and use it as the grouping value in the Reflection, DATE. If Dremio should use the original value for grouping, NORMAL.
Valid Values: DATE, NORMAL
Example: DATE
Parameters of Objects in the measureFields Array
name Body String
Name of the field from the anchor dataset that you expect to use in calculations. Fields of types LIST, MAP, and UNION are not valid measureFields.
Example: passenger_count
measureTypeList Body Array of String
Types of calculations for which Dremio should use the specified measure field. The calculations must be valid for the specified field (for example, SUM is not valid for a timestamp field like pickup_datetime).
Valid Values: APPROX_COUNT_DISTINCT, MIN, MAX, UNKNOWN, SUM, COUNT
Example:
[
"COUNT",
"SUM"
]
Parameters of Objects in the distributionFields Array
name Body String
Name of the field from the anchor dataset to use for co-locating and co-partitioning data from multiple datasets across nodes. Every field listed as a distribution field must also be listed as a dimension field.
Example: trip_distance_mi
Parameters of Objects in the partitionFields Array
name Body String
Name of the field from the anchor dataset on which you want to be able to partition rows. Every field listed as a partition field must also be listed as a dimension field. If you list a field as a partition field, you cannot list the same field as a sort field in the same Reflection.
Example: pickup_datetime
Parameters of Objects in the sortFields Array
name Body String
Name of the field from the anchor dataset to use for sorting in the Reflection. Every field listed as a sort field must also be listed as a dimension field. If you list a field as a sort field, you cannot list the same field as a partition field in the same Reflection.
Example: trip_distance_mi
Example
Requestcurl -X PUT "https://api.dremio.cloud/v0/projects/$PROJECT_ID/reflection/$REFLECTION_ID" \
-H "Authorization: Bearer $DREMIO_TOKEN" \
-H 'Content-Type: application/json' \
--data-raw '{
"id": "836eae91-306e-487b-a687-31c999653a86",
"type": "AGGREGATION",
"name": "New Aggregation Reflection",
"tag": "ff9f4477-e44d-4995-a76a-70edbe572c56",
"datasetId": "81e2ad31-a119-447d-a831-085831e505be",
"enabled": false,
"dimensionFields": [
{
"name": "pickup_datetime",
"granularity": "DATE"
},
{
"name": "passenger_count",
"granularity": "DATE"
},
{
"name": "total_amount",
"granularity": "DATE"
},
{
"name": "trip_distance_mi",
"granularity": "DATE"
}
],
"measureFields": [
{
"name": "passenger_count",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "trip_distance_mi",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "fare_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "tip_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "total_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
}
],
"distributionFields": [
{
"name": "trip_distance_mi"
},
{
"name": "total_amount"
}
],
"partitionFields": [
{
"name": "pickup_datetime"
},
{
"name": "passenger_count"
}
],
"sortFields": [
{
"name": "trip_distance_mi"
}
],
"entityType": "reflection"
}'
{
"id": "26526a2e-3edb-4af0-a24f-8e9e2dcfa82a",
"type": "AGGREGATION",
"name": "New Aggregation Reflection",
"tag": "59dcb49e-ed20-49b3-9e54-b9d02b4411d4",
"createdAt": "2023-01-30T14:35:19.192Z",
"updatedAt": "2023-01-30T14:35:19.192Z",
"datasetId": "81e2ad31-a119-447d-a831-085831e505be",
"currentSizeBytes": 8808,
"totalSizeBytes": 8808,
"enabled": false,
"status": {
"config": "OK",
"refresh": "SCHEDULED",
"availability": "AVAILABLE",
"combinedStatus": "CAN_ACCELERATE",
"failureCount": 0,
"lastDataFetch": "2023-02-03T18:40:37.655Z",
"expiresAt": "3023-01-30T14:35:01.180Z"
},
"dimensionFields": [
{
"name": "pickup_datetime",
"granularity": "DATE"
},
{
"name": "passenger_count",
"granularity": "DATE"
},
{
"name": "total_amount",
"granularity": "DATE"
},
{
"name": "trip_distance_mi",
"granularity": "DATE"
}
],
"measureFields": [
{
"name": "passenger_count",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "trip_distance_mi",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "fare_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "tip_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "total_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
}
],
"distributionFields": [
{
"name": "trip_distance_mi"
},
{
"name": "total_amount"
}
],
"partitionFields": [
{
"name": "pickup_datetime"
},
{
"name": "passenger_count"
}
],
"sortFields": [
{
"name": "trip_distance_mi"
}
],
"partitionDistributionStrategy": "CONSOLIDATED",
"canView": true,
"canAlter": true,
"entityType": "reflection"
}
Response Status Codes
200 OK
401 Unauthorized
404 Not Found
409 Conflict
500 Internal Server Error
Delete a Reflection
Method and URLDELETE /v0/projects/{project_id}/reflection/{id}
Parameters
project_id Path String (UUID)
id Path String (UUID)
UUID for the Reflection that you want to delete.
Example
Requestcurl -X DELETE "https://api.dremio.cloud/v0/projects/$PROJECT_ID/reflection/$REFLECTION_ID"
-H 'Authorization: Bearer <access_token>' \
-H 'Content-Type:application/json'
A successful request returns an empty response body with HTTP status 204 No Content.
Response Status Codes
204 No Content
401 Unauthorized
404 Not Found
405 Method Not Allowed