Skip to main content
Version: current [24.3.x]

Reflection

Use the Reflection API to retrieve a list of raw and aggregation reflections, retrieve individual reflections, and create, update, and delete reflections.

A reflection is an optimized materialization of source data or a query, similar to a materialized view, that is derived from an existing table or view. The query optimizer can accelerate queries by using one or more reflections to partially or entirely satisfy the queries rather than running queries against the raw data in the data source that underlies the table or view.

Reflection Object (Raw Reflection)
{
"id": "7a380a24-3b63-436c-9ea0-63cb534cc404",
"type": "RAW",
"name": "Raw Reflection",
"tag": "ureIY76RT7Y=",
"createdAt": "2023-01-30T14:11:43.826Z",
"updatedAt": "2023-01-30T14:11:43.826Z",
"datasetId": "tk973df7-ddf7-4d1e-fa9e-bccf28ae253f",
"currentSizeBytes": 4393709246,
"totalSizeBytes": 4393709246,
"enabled": true,
"arrowCachingEnabled": false,
"status": {
"config": "OK",
"refresh": "SCHEDULED",
"availability": "AVAILABLE",
"combinedStatus": "CAN_ACCELERATE",
"failureCount": 0,
"lastDataFetch": "2023-01-30T14:11:51.801Z",
"expiresAt": "2023-01-30T17:11:51.801Z"
},
"displayFields": [
{
"name": "pickup_datetime"
},
{
"name": "passenger_count"
},
{
"name": "trip_distance_mi"
},
{
"name": "fare_amount"
},
{
"name": "tip_amount"
},
{
"name": "total_amount"
}
],
"distributionFields": [
{
"name": "trip_distance_mi"
}
],
"partitionFields": [
{
"name": "passenger_count"
}
],
"sortFields": [
{
"name": "pickup_datetime"
}
],
"partitionDistributionStrategy": "CONSOLIDATED",
"canView": true,
"canAlter": true,
"entityType": "reflection"
}
Reflection Object (Aggregation Reflection)
{
"id": "95dda9dd-2371-467f-b68d-fc4c5ea57a8b",
"type": "AGGREGATION",
"name": "Aggregation Reflection",
"tag": "ZpzGgxw2l04=",
"createdAt": "2022-07-05T19:19:40.244Z",
"updatedAt": "2023-01-10T17:12:40.244Z",
"datasetId": "df99ab32-c2d4-4d1c-9e91-2c8be861bb8a",
"currentSizeBytes": 18639885,
"totalSizeBytes": 142639924,
"enabled": true,
"arrowCachingEnabled": false,
"status": {
"config": "OK",
"refresh": "SCHEDULED",
"availability": "AVAILABLE",
"combinedStatus": "CAN_ACCELERATE",
"failureCount": 0,
"lastDataFetch": "2023-01-10T17:12:40.244Z",
"expiresAt": "3022-07-05T19:19:40.244Z"
},
"dimensionFields": [
{
"name": "pickup_date"
},
{
"name": "pickup_datetime",
"granularity": "DATE"
},
{
"name": "dropoff_date"
},
{
"name": "dropoff_datetime",
"granularity": "DATE"
},
{
"name": "passenger_count"
},
{
"name": "total_amount"
},
{
"name": "trip_distance_mi"
}
],
"measureFields": [
{
"name": "passenger_count",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "trip_distance_mi",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "fare_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "surcharge",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "tip_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "total_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
}
],
"distributionFields": [
{
"name": "trip_distance_mi"
},
{
"name": "total_amount"
}
],
"partitionFields": [
{
"name": "dropoff_date"
},
{
"name": "passenger_count"
}
],
"sortFields": [
{
"name": "trip_distance_mi"
}
],
"partitionDistributionStrategy": "CONSOLIDATED",
"canView": true,
"canAlter": true,
"entityType": "reflection"
}

Reflection Attributes

id String (UUID)

Unique identifier of the reflection.

Example: 95dda9dd-2371-467f-b68d-fc4c5ea57a8b


type String

Reflection type. For more information, read Types of Reflections.

Enum: RAW, AGGREGATION

Example: AGGREGATION


name String

User-provided name for the reflection. For reflections created in the Dremio UI, if the user did not provide a name, the default values are Raw Reflection and Aggregation Reflection (automatically assigned based on the reflection type).

Example: Aggregation Reflection


tag String

Unique identifier of the reflection instance. Dremio changes the tag whenever the reflection changes and uses the tag to ensure that PUT requests apply to the most recent version of the reflection.

Example: ZpzGgxw2l04=


createdAt String

Date and time that the reflection was created, in UTC format.

Example: 2022-07-05T19:19:40.244Z


updatedAt String

Date and time that the reflection was last updated, in UTC format.

Example: 2023-01-10T17:12:40.244Z


datasetId String (UUID)

Unique identifier of the anchor dataset that is associated with the reflection.

Example: df99ab32-c2d4-4d1c-9e91-2c8be861bb8a


currentSizeBytes Integer

Data size of the latest reflection job (if one exists), in bytes.

Example: 18639885


totalSizeBytes Integer

Data size of all reflection jobs that have not been pruned (if any exist), in bytes.

Example: 142639924


enabled Boolean

If the reflection is available for accelerating queries, the value is true. Otherwise, the value is false.

Example: true


arrowCachingEnabled Boolean

If Dremio converts data from the reflection's Parquet files to Apache Arrow format when copying that data to executor nodes, the value is true. Otherwise, the value is false.

Example: false


status Object

Information about the status of the reflection.

Example: {"config": "OK","refresh": "SCHEDULED","availability": "AVAILABLE","combinedStatus": "CAN_ACCELERATE","failureCount": 0,"lastDataFetch": "2023-01-10T17:12:40.244Z","expiresAt": "3022-07-05T19:19:40.244Z"}


displayFields Array of Object

Information about the fields displayed from the anchor dataset. Each displayFields object contains one attribute: name. Valid only for raw reflections.

Example: [{"name": "pickup_datetime"},{"name": "passenger_count"},{"name": "trip_distance_mi"},{"name": "fare_amount"},{"name": "tip_amount"},{"name": "total_amount"}]


dimensionFields Array of Object

Information about the dimension fields from the anchor dataset used in the reflection. Dimension fields are the fields you expect to group by when analyzing data. Each dimensionFields object contains two attributes: name and granularity. Valid only for aggregation reflections.

Example: [{"name": "pickup_date","granularity": "DATE"},{"name": "pickup_datetime","granularity": "DATE"},{"name": "dropoff_date","granularity": "DATE"},{"name": "dropoff_datetime","granularity": "DATE"},{"name": "passenger_count","granularity": "DATE"},{"name": "total_amount","granularity": "DATE"}]


measureFields Array of Object

Information about the measure fields from the anchor dataset used in the reflection. Measure fields are the fields you expect to use for calculations when analyzing the data. Each measureFields object contains two attributes: name and measureTypeList. Valid only for aggregation reflections.

Example: [{"name": "passenger_count","measureTypeList": ["SUM", "COUNT"]},{"name": "trip_distance_mi","measureTypeList": ["SUM", "COUNT"]},{"name": "fare_amount","measureTypeList": ["SUM", "COUNT"]},{"name": "surcharge","measureTypeList": ["SUM", "COUNT"]},{"name": "tip_amount","measureTypeList": ["SUM", "COUNT"]},{"name": "total_amount","measureTypeList": ["SUM", "COUNT"]}]


distributionFields Array of Object

Information about the distribution fields from the anchor dataset used in the reflection. Distribution fields allow data from multiple datasets to be co-located and co-partitioned across nodes to minimize data movement during join operations. Each distributionFields object contains one attribute: name.

Example: [{"name": "trip_distance_mi"},{"name": "total_amount"}]


partitionFields Array of Object

Information about the fields from the anchor dataset used to partition data in the reflection. Each field name is listed as an individual object. For more information, read Horizontally Partition Reflections that Have Many Rows.

Example: [{"name": "dropoff_date"},{"name": "passenger_count"}]


sortFields Array of Object

Information about the fields from the anchor dataset used for sorting in the reflection. Each sortFields object contains one attribute: name. For more information, read Sort Reflections on High-Cardinality Fields.

Example: [{"name": "trip_distance_mi"}]


partitionDistributionStrategy String

Method used to optimize data compression when executing reflections. CONSOLIDATED means Dremio minimizes the number of files produced. The query threads pool the data and ensure that the fewest number of files are written to the reflection store. Optimizing for a smaller number of files generally improves read performance because users can perform fewer searches for a given query. STRIPED means Dremio minimizes the time required to refresh the reflection. Each final-stage query thread opens its own writers to write the data, which can result in many small files if each query thread contains a small amount of data.

Enum: CONSOLIDATED, STRIPED

Example: CONSOLIDATED


canView Boolean

If you can view reflections on all datasets of a source, system, space, or folder, the value is true. Otherwise, the value is false.

Example: true


canAlter Boolean

If you can create, edit, and view reflections on all datasets of a source, system, space, or folder, the value is true. Otherwise, the value is false.

Example: true


entityType String

Type of the object. For reflection objects, the entityType is reflection.

Example: reflection

Attributes of the status Object

config String

Status of the reflection configuration. If the value is OK, the reflection configuration is free of errors. If the value is INVALID, the reflection configuration contains one or more errors.

Enum: OK, INVALID

Example: OK


refresh String

Status of the reflection refresh.

  • GIVEN_UP: Dremio attempted to refresh the reflection multiple times, but each attempt has failed and Dremio will not make further attempts.
  • MANUAL: Refresh period is set to 0, so you must use the Dremio UI to manually refresh the reflection.
  • RUNNING: Dremio is currently refreshing the reflection.
  • SCHEDULED: The reflection refreshes according to a schedule.

Enum: GIVEN_UP, MANUAL, RUNNING, SCHEDULED

Example: SCHEDULED


availability String

Status of the reflection's availability for accelerating queries.

Enum: NONE, INCOMPLETE, EXPIRED, AVAILABLE

Example: AVAILABLE


combinedStatus String

Status of the reflection based on a combination of config, refresh, and availability.

  • CAN_ACCELERATE: The reflection is fully functional.
  • CAN_ACCELERATE_WITH_FAILURES: The most recent refresh failed to obtain a status, but Dremio still has a valid materialization.
  • CANNOT_ACCELERATE_MANUAL: The reflection is unable to accelerate any queries, and the Never Refresh option is selected for the refresh policy.
  • CANNOT_ACCELERATE_SCHEDULED: The reflection is currently unable to accelerate any queries, but it has been scheduled for a refresh at a future time.
  • DISABLED: The reflection has been manually disabled.
  • EXPIRED: The reflection has expired and cannot be used.
  • FAILED: The attempt to refresh the reflection has failed, typically three times in a row. The reflection is still usable.
  • INVALID: The reflection is invalid because the underlying dataset has changed.
  • INCOMPLETE: One or more pseudo-distributed file system (PDFS) nodes that contain materialized files are down (PFDS is supported for v21 and earlier). Only partial data is available. Configurations that use the Hadoop Distributed File System (HDFS) to store reflections should not experience incomplete status.
  • REFRESHING: The reflection is currently being refreshed.

Example: CAN_ACCELERATE


failureCount Integer

Number of times that an attempt to refresh the reflection failed.

Example: 0


lastDataFetch String

Date and time that the reflection data was last refreshed, in UTC format. If the reflection is running, failing, or disabled, the lastDataFetch value is 1969-12-31T23:59:59.999Z.

Example: 2023-01-10T17:12:40.244Z


expiresAt String

Date and time that the reflection expires, in UTC format. If the reflection is running, failing, or disabled, the lastDataFetch value is 1969-12-31T23:59:59.999Z.

Example: 3022-07-05T19:19:40.244Z


Attributes of the displayFields Object

name String

Name of the field from the anchor dataset that is displayed in the raw reflection.

Example: passenger_count

Attributes of the dimensionFields Object

name String

Name of the field from the anchor dataset that is configured as a dimension for the reflection.

Example: pickup_date


granularity String

Grouping used for the dimension field. When timestamp and date fields are configured as dimensions, Dremio can automatically extract and use the day-level date value (DATE) or use the field's original value (NORMAL).

Enum: DATE, NORMAL

Example: DATE

Attributes of the measureFields Object

name String

Name of the field from the anchor dataset that is configured as a measure for the reflection.

Example: passenger_count


measureTypeList Array of String

Types of calculations for which Dremio uses the specified measure field.

Enum: APPROX_COUNT_DISTINCT, MIN, MAX, UNKNOWN, SUM, COUNT

Example: ["SUM","COUNT"]

Attributes of the distributionFields Object

name String

Name of the field from the anchor dataset that is used for co-locating and co-partitioning data from multiple datasets across nodes.

Example: trip_distance_mi

Attributes of the partitionFields Object

name String

Name of the field from the anchor dataset on which the rows in the reflection are to be partitioned. If a column is listed as a partition column, it cannot also be listed as a sort column for the same reflection. In aggregation reflections, each column specified as a partition column or used in transform must also be listed as a dimension column. In raw reflections, each column specified as a partition column or used in transform must also be listed as a display column.

Example: trip_distance_mi


transform Object

The type of partition transform that is applied. The value is an enum. The types are:

IDENTITY: Creates one partition per value. This is the default transform. If no transform is specified for a field named by the name property, an IDENTITY transform is performed.

IDENTITY Example
{
"name": "passenger_count",
"transform": {
"type": "IDENTITY"
}
}

YEAR: Partitions by year. The field must use the TIMESTAMP or DATE data type.

YEAR Example
{
"name": "pickup_datetime",
"transform": {
"type": "YEAR"
}
}

MONTH: Partitions by month. The field must use the TIMESTAMP or DATE data type.

MONTH Example
{
"name": "pickup_datetime",
"transform": {
"type": "MONTH"
}
}

DAY: Partitions on the equivalent of dateint. The field must use the TIMESTAMP or DATE data type.

DAY Example
{
"name": "pickup_datetime",
"transform": {
"type": "DAY"
}
}

HOUR: Partitions on the equivalent of dateint and hour. The field must use the TIMESTAMP data type.

HOUR Example
{
"name": "pickup_datetime",
"transform": {
"type": "HOUR"
}
}

BUCKET: Partitions data into the number of partitions specified by an integer. For example, if the integer value N is specified, the data is partitioned into N, or (0 to (N-1)), partitions. The partition in which an individual row is stored is determined by hashing the field value and then calculating <hash_value> mod N. If the result is 0, the row is placed in partition 0; if the result is 1, the row is placed in partition 1; and so on.

This value must be followed by a bucketTransform object. This object takes one property: bucketCount. This property takes an integer value.

BUCKET Example
{
"name": "pickup_datetime",
"transform": {
"type": "BUCKET",
"bucketTransform": {
"bucketCount": 1000
}
}
}

TRUNCATE: If the specified field uses the string data type, truncates strings to a maximum of the number of characters specified by an integer. For example, suppose the specified transform is truncate(1, stateUS). A value of CA is truncated to C, and the row is placed in partition C. A value of CO is also truncated to C, and the row is also placed in partition C.

If the specified field uses the integer or long data type, truncates field values in the following way: For any truncate(L, col), truncates the field value to the biggest multiple of L that is smaller than the field value. For example, suppose the specified transform is truncate(10, intField). A value of 1 is truncated to 0 and the row is placed in the partition 0. A value of 247 is truncated to 240 and the row is placed in partition 240. If the transform is truncate(3, intField), a value of 13 is truncated to 12 and the row is placed in partition 12. A value of 255 is not truncated, because it is divisble by 3, and the row is placed in partition 255. This value must be followed by a truncateTransform object.

This object takes one property: truncateLength. This property takes an integer value.

note

The truncate transform does not change field values. It uses field values to calculate the correct partitions in which to place rows.

TRUNCATE Example
{ 
"name": "pickup_hour",
"transform": {
"type": "TRUNCATE",
"truncateTransform": {
"truncateLength": 3
}
}
}

Attributes of the sortFields Object

name String

Name of the field from the anchor dataset that is used for sorting in the reflection.

Example: dropoff_date

Creating a Reflection

Create a new reflection.

Method and URL
POST /api/v3/reflection

Parameters

type Body   String

Reflection type. For more information, read Types of Reflections.

Enum: RAW, AGGREGATION

Example: AGGREGATION


name Body   String

Name to use for the reflection.

Example: New Aggregation Reflection


datasetId Body   String (UUID)

Unique identifier of the anchor dataset to associate with the reflection.

Example: df99ab32-c2d4-4d1c-9e91-2c8be861bb8a


enabled Body   Boolean

If the reflection should be available for accelerating queries, set to true. Otherwise, set to false.

Example: true


arrowCachingEnabled Body   Boolean   Optional

If Dremio should convert data from the reflection's Parquet files to Apache Arrow format when copying that data to executor nodes, set to true. Otherwise, set to false (default).

Example: false


displayFields Body   Array of Object   Optional

Information about the fields to display from the anchor dataset. The displayfields array must list every field in the anchor dataset or the reflection fails. Each displayFields object contains one attribute: name. Valid only for raw reflections.

Example: [{"name": "pickup_datetime"},{"name": "passenger_count"},{"name": "trip_distance_mi"},{"name": "fare_amount"},{"name": "tip_amount"},{"name": "total_amount"}]


dimensionFields Body   Array of Object   Optional

Information about the dimension fields from the anchor dataset to use in the reflection. Dimension fields are the fields you expect to group by when analyzing data. Each dimensionFields object contains two attributes: name and granularity. Valid only for aggregation reflections.

Example: [{"name": "pickup_datetime","granularity": "DATE"},{"name": "passenger_count","granularity": "DATE"},{"name": "total_amount","granularity": "DATE"},{"name": "trip_distance_mi","granularity": "DATE"}]


measureFields Body   Array of Object   Optional

Information about the measure fields from the anchor dataset to use in the reflection. Measure fields are the fields you expect to use for calculations when analyzing the data. Each measureFields object contains two attributes: name and measureTypeList. Valid only for aggregation reflections.

Example: [{"name": "passenger_count","measureTypeList": ["SUM", "COUNT"]},{"name": "trip_distance_mi","measureTypeList": ["SUM", "COUNT"]},{"name": "fare_amount","measureTypeList": ["SUM", "COUNT"]},{"name": "tip_amount","measureTypeList": ["SUM", "COUNT"]},{"name": "total_amount","measureTypeList": ["SUM", "COUNT"]}]


distributionFields Body   Array of Object   Optional

Information about the distribution fields from the anchor dataset to use for co-locating and co-partitioning data from multiple datasets across nodes. Each distributionFields object contains one attribute: name.

Example: [{"name": "trip_distance_mi"},{"name": "total_amount"}]


partitionFields Body   Array of Object   Optional

Information about the fields from the anchor dataset to use to partition data in the reflection. Each field name is listed as an individual object. For more information, read Horizontally Partition Reflections that Have Many Rows.

Example: [{"name": "pickup_datetime"},{"name": "passenger_count"}]


sortFields Body   Array of Object   Optional

Information about the fields from the anchor dataset to use for sorting in the reflection. Each sortFields object contains one attribute: name. For more information, read Sort Reflections on High-Cardinality Fields.

Example: [{"name": "trip_distance_mi"}]


partitionDistributionStrategy Body   String   Optional

Method to use to optimize data compression when executing reflections. If set to CONSOLIDATED (default), Dremio minimizes the number of files produced. If set to STRIPED, Dremio minimizes the time required to refresh the reflection.

Enum: CONSOLIDATED, STRIPED

Example: CONSOLIDATED


canView Body   Boolean   Optional

To view reflections on all datasets of a source, system, space, or folder, set to true (default). Otherwise, set to false.

Example: true


canAlter Body   Boolean   Optional

To create, edit, and view reflections on all datasets of a source, system, space, or folder, set to true (default). Otherwise, set to false.

Example: true


entityType Body   String   Optional

Type of the object. For reflection objects, the entityType is reflection.

Example: reflection

Parameters of the displayFields Object

name Body   String

Name of the field to display from the anchor dataset.

Example: "name": "pickup_datetime"

Parameters of the dimensionFields Object

name Body   String

Name of the field from the anchor dataset to configure as a dimension for the reflection.

Example: "name": "pickup_datetime"


granularity Body   String

Grouping to use for the dimension field. If Dremio should automatically extract the day-level date value and use it as the grouping value in the reflection, DATE. If Dremio should use the original value for grouping, NORMAL.

Enum: DATE, NORMAL

Example: "granularity": "DATE"

Parameters of the measureFields Object

name Body   String

Name of the field from the anchor dataset that you expect to use in calculations. Fields of types LIST, MAP, and UNION are not valid measureFields.

Example: "name": "passenger_count"


measureTypeList Body   Array of String

Name of the field from the anchor dataset to use for co-locating and co-partitioning data from multiple datasets across nodes. In aggregation reflections, every field listed as a distribution field must also be listed as a dimension field.

Enum: APPROX_COUNT_DISTINCT, MIN, MAX, UNKNOWN, SUM, COUNT

Example: ["SUM", "COUNT"]

Parameters of the distributionFields Object

name Body   String

Name of the field from the anchor dataset to use for co-locating and co-partitioning data from multiple datasets across nodes. In aggregation reflections, every field listed as a distribution field must also be listed as a dimension field.

Example: "name": "trip_distance_mi"

Parameters of the partitionFields Object

name Body   String

Name of the field from the anchor dataset on which you want to be able to partition rows. If you are creating an aggregation reflection, every field listed as a partition field must also be listed as a dimension field. If you list a field as a partition field, you cannot list the same field as a sort field in the same reflection.

Example: "name": "pickup_datetime"


transform Object

The type of partition transform that is applied. The value is an enum. The types are:

IDENTITY: Creates one partition per value. This is the default transform. If no transform is specified for a field named by the name property, an IDENTITY transform is performed.

IDENTITY Example
{
"name": "passenger_count",
"transform": {
"type": "IDENTITY"
}
}

YEAR: Partitions by year. The field must use the TIMESTAMP or DATE data type.

YEAR Example
{
"name": "pickup_datetime",
"transform": {
"type": "YEAR"
}
}

MONTH: Partitions by month. The field must use the TIMESTAMP or DATE data type.

MONTH Example
{
"name": "pickup_datetime",
"transform": {
"type": "MONTH"
}
}

DAY: Partitions on the equivalent of dateint. The field must use the TIMESTAMP or DATE data type.

DAY Example
{
"name": "pickup_datetime",
"transform": {
"type": "DAY"
}
}

HOUR: Partitions on the equivalent of dateint and hour. The field must use the TIMESTAMP data type.

HOUR Example
{
"name": "pickup_datetime",
"transform": {
"type": "HOUR"
}
}

BUCKET: Partitions data into the number of partitions specified by an integer. For example, if the integer value N is specified, the data is partitioned into N, or (0 to (N-1)), partitions. The partition in which an individual row is stored is determined by hashing the field value and then calculating <hash_value> mod N. If the result is 0, the row is placed in partition 0; if the result is 1, the row is placed in partition 1; and so on.

This value must be followed by a bucketTransform object. This object takes one property: bucketCount. This property takes an integer value.

BUCKET Example
{
"name": "pickup_datetime",
"transform": {
"type": "BUCKET",
"bucketTransform": {
"bucketCount": 1000
}
}
}

TRUNCATE: If the specified field uses the string data type, truncates strings to a maximum of the number of characters specified by an integer. For example, suppose the specified transform is truncate(1, stateUS). A value of CA is truncated to C, and the row is placed in partition C. A value of CO is also truncated to C, and the row is also placed in partition C.

If the specified field uses the integer or long data type, truncates field values in the following way: For any truncate(L, col), truncates the field value to the biggest multiple of L that is smaller than the field value. For example, suppose the specified transform is truncate(10, intField). A value of 1 is truncated to 0 and the row is placed in the partition 0. A value of 247 is truncated to 240 and the row is placed in partition 240. If the transform is truncate(3, intField), a value of 13 is truncated to 12 and the row is placed in partition 12. A value of 255 is not truncated, because it is divisble by 3, and the row is placed in partition 255. This value must be followed by a truncateTransform object.

This object takes one property: truncateLength. This property takes an integer value.

note

The truncate transform does not change field values. It uses field values to calculate the correct partitions in which to place rows.

TRUNCATE Example
{ 
"name": "pickup_hour",
"transform": {
"type": "TRUNCATE",
"truncateTransform": {
"truncateLength": 3
}
}
}

Parameters of the sortFields Object

name Body   String

Name of the field from the anchor dataset to use for sorting in the reflection. Every field listed as a sort field must also be listed as a dimension field. If you list a field as a sort field, you cannot list the same field as a partition field in the same reflection.

Example: "name": "trip_distance_mi"

Example Request
curl -X POST 'https://{hostname}/api/v3/reflection/' \
--header 'Authorization: Bearer <PersonalAccessToken>' \
--header 'Content-Type: application/json' \
--data-raw '{
"type": "AGGREGATION",
"name": "New Aggregation Reflection",
"datasetId": "gc870df7-ddf7-4d1e-bb9e-beef28ae773f",
"enabled": true,
"arrowCachingEnabled": false,
"dimensionFields": [
{
"name": "pickup_datetime",
"granularity": "DATE"
},
{
"name": "passenger_count"
},
{
"name": "total_amount"
},
{
"name": "trip_distance_mi"
}
],
"measureFields": [
{
"name": "passenger_count",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "trip_distance_mi",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "fare_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "tip_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "total_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
}
],
"distributionFields": [
{
"name": "trip_distance_mi"
},
{
"name": "total_amount"
}
],
"partitionFields": [
{
"name": "pickup_datetime"
},
{
"name": "passenger_count"
}
],
"sortFields": [
{
"name": "trip_distance_mi"
}
],
"entityType": "reflection"
}'
Example Response
{
"id": "836eae91-306e-487b-a687-31c999653a86",
"type": "AGGREGATION",
"name": "New Aggregation Reflection",
"tag": "sEHieiuinqE=",
"createdAt": "2023-01-30T14:30:24.311Z",
"updatedAt": "2023-01-30T14:30:24.311Z",
"datasetId": "gc870df7-ddf7-4d1e-bb9e-beef28ae773f",
"currentSizeBytes": 0,
"totalSizeBytes": 0,
"enabled": true,
"arrowCachingEnabled": false,
"status": {
"config": "OK",
"refresh": "SCHEDULED",
"availability": "NONE",
"combinedStatus": "CANNOT_ACCELERATE_SCHEDULED",
"failureCount": 0,
"lastDataFetch": "1969-12-31T23:59:59.999Z",
"expiresAt": "1969-12-31T23:59:59.999Z"
},
"dimensionFields": [
{
"name": "pickup_datetime",
"granularity": "DATE"
},
{
"name": "passenger_count"
},
{
"name": "total_amount"
},
{
"name": "trip_distance_mi"
}
],
"measureFields": [
{
"name": "passenger_count",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "trip_distance_mi",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "fare_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "tip_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "total_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
}
],
"distributionFields": [
{
"name": "trip_distance_mi"
},
{
"name": "total_amount"
}
],
"partitionFields": [
{
"name": "pickup_datetime"
},
{
"name": "passenger_count"
}
],
"sortFields": [
{
"name": "trip_distance_mi"
}
],
"partitionDistributionStrategy": "CONSOLIDATED",
"canView": true,
"canAlter": true,
"entityType": "reflection"
}

Response Status Codes

200   OK

401   Unauthorized

404   Not Found

405   Method Not Allowed

500   Internal Server Error

Retrieving All Reflections Enterprise

Retrieve a list of reflection objects that includes all raw and aggregation reflections in the Dremio instance.

Method and URL
GET /api/v3/reflection
Example Request
curl -X GET 'https://{hostname}/api/v3/reflection/' \
--header 'Authorization: Bearer <PersonalAccessToken>' \
--header 'Content-Type: application/json'

In the response for a request to retrieve all raw and aggregation reflections, the reflection objects are wrapped with a data array. Each object in the data array represents one reflection.

Example Response
{
"data": [
{
"id": "95dda9dd-2371-467f-b68d-fc4c5ea57a8b",
"type": "AGGREGATION",
"name": "Aggregation Reflection",
"tag": "ZpzGgxw2l04=",
"createdAt": "2022-07-05T19:19:40.244Z",
"updatedAt": "2023-01-10T17:12:40.244Z",
"datasetId": "df99ab32-c2d4-4d1c-9e91-2c8be861bb8a",
"currentSizeBytes": 18639885,
"totalSizeBytes": 142639924,
"enabled": true,
"arrowCachingEnabled": false,
"status": {
"config": "OK",
"refresh": "SCHEDULED",
"availability": "AVAILABLE",
"combinedStatus": "CAN_ACCELERATE",
"failureCount": 0,
"lastDataFetch": "2023-01-10T17:12:40.244Z",
"expiresAt": "3022-07-05T19:19:40.244Z"
},
"dimensionFields": [
{
"name": "pickup_date"
},
{
"name": "pickup_datetime",
"granularity": "DATE"
},
{
"name": "dropoff_date"
},
{
"name": "dropoff_datetime",
"granularity": "DATE"
},
{
"name": "passenger_count"
},
{
"name": "total_amount"
},
{
"name": "trip_distance_mi"
}
],
"measureFields": [
{
"name": "passenger_count",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "trip_distance_mi",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "fare_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "surcharge",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "tip_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "total_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
}
],
"distributionFields": [
{
"name": "trip_distance_mi"
},
{
"name": "total_amount"
}
],
"partitionFields": [
{
"name": "dropoff_date"
},
{
"name": "passenger_count"
}
],
"sortFields": [
{
"name": "trip_distance_mi"
}
],
"partitionDistributionStrategy": "CONSOLIDATED",
"canView": true,
"canAlter": true,
"entityType": "reflection"
},
{
"id": "14f22052-cbb3-4d5d-8bbc-6154cca98e49",
"type": "RAW",
"name": "listings",
"tag": "XAy4ccVFXO4=",
"createdAt": "2022-07-12T16:45:35.249Z",
"updatedAt": "2022-07-12T16:45:35.249Z",
"datasetId": "7707981c-cb33-42bc-a048-d27a8915f468",
"currentSizeBytes": 0,
"totalSizeBytes": 0,
"enabled": true,
"arrowCachingEnabled": true,
"status": {
"config": "OK",
"refresh": "MANUAL",
"availability": "NONE",
"combinedStatus": "CANNOT_ACCELERATE_MANUAL",
"failureCount": 0,
"lastDataFetch": "1969-12-31T23:59:59.999Z",
"expiresAt": "1969-12-31T23:59:59.999Z"
},
"displayFields": [
{
"name": "id"
}
],
"partitionDistributionStrategy": "CONSOLIDATED",
"canView": true,
"canAlter": true,
"entityType": "reflection"
},
{
"id": "6c209200-b522-4f81-bbe0-d10668c7752c",
"type": "AGGREGATION",
"name": "Aggregation Reflection",
"tag": "SQeEAG3d6DA=",
"createdAt": "2021-09-29T15:47:44.806Z",
"updatedAt": "2021-09-29T15:47:44.806Z",
"datasetId": "746f867a-c27c-4711-bb8c-99546a4c25e0",
"currentSizeBytes": 0,
"totalSizeBytes": 1675978,
"enabled": true,
"arrowCachingEnabled": false,
"status": {
"config": "OK",
"refresh": "GIVEN_UP",
"availability": "NONE",
"combinedStatus": "FAILED",
"failureCount": 3,
"lastDataFetch": "1969-12-31T23:59:59.999Z",
"expiresAt": "1969-12-31T23:59:59.999Z"
},
"dimensionFields": [
{
"name": "passenger_count"
},
{
"name": "pickup_datetime",
"granularity": "DATE"
}
],
"measureFields": [
{
"name": "trip_distance_mi",
"measureTypeList": [
"COUNT",
"SUM"
]
},
{
"name": "total_amount",
"measureTypeList": [
"COUNT",
"SUM"
]
},
{
"name": "tip_amount",
"measureTypeList": [
"COUNT",
"SUM"
]
},
{
"name": "fare_amount",
"measureTypeList": [
"COUNT",
"SUM"
]
}
],
"partitionDistributionStrategy": "CONSOLIDATED",
"canView": true,
"canAlter": true,
"entityType": "reflection"
},
{
"id": "c5c5b282-ffea-4a34-835f-cc591584412b",
"type": "AGGREGATION",
"name": "Test reflection",
"tag": "lMxFcc2qjgE=",
"createdAt": "2021-10-11T18:44:27.064Z",
"updatedAt": "2021-10-11T18:44:27.064Z",
"datasetId": "316531b8-3c56-42f2-b05f-81f228ef3162",
"currentSizeBytes": 0,
"totalSizeBytes": 0,
"enabled": true,
"arrowCachingEnabled": false,
"status": {
"config": "OK",
"refresh": "MANUAL",
"availability": "NONE",
"combinedStatus": "CANNOT_ACCELERATE_MANUAL",
"failureCount": 0,
"lastDataFetch": "1969-12-31T23:59:59.999Z",
"expiresAt": "1969-12-31T23:59:59.999Z"
},
"dimensionFields": [
{
"name": "passenger_count"
}
],
"measureFields": [
{
"name": "trip_distance_mi",
"measureTypeList": [
"COUNT",
"SUM"
]
},
{
"name": "total_amount",
"measureTypeList": [
"COUNT",
"SUM"
]
},
{
"name": "tip_amount",
"measureTypeList": [
"COUNT",
"SUM"
]
},
{
"name": "fare_amount",
"measureTypeList": [
"COUNT",
"SUM"
]
}
],
"partitionDistributionStrategy": "CONSOLIDATED",
"canView": true,
"canAlter": true,
"entityType": "reflection"
}
],
"canAlterReflections": true
}

Response Status Codes

200   OK

401   Unauthorized

404   Not Found

500   Internal Server Error

Retrieving a Reflection

Retrieve the specified reflection.

Method and URL
GET /api/v3/reflection/{id}

Parameters

id Path   String (UUID)

Unique identifier of the reflection that you want to retrieve.

Example: 95dda9dd-2371-467f-b68d-fc4c5ea57a8b

Example Request
curl -X GET 'https://{hostname}/api/v3/reflection/95dda9dd-2371-467f-b68d-fc4c5ea57a8b'
--header 'Authorization: Bearer <PersonalAccessToken>' \
--header 'Content-Type: application/json'
Example Response
{
"id": "95dda9dd-2371-467f-b68d-fc4c5ea57a8b",
"type": "AGGREGATION",
"name": "Aggregation Reflection",
"tag": "ZpzGgxw2l04=",
"createdAt": "2022-07-05T19:19:40.244Z",
"updatedAt": "2023-01-10T17:12:40.244Z",
"datasetId": "df99ab32-c2d4-4d1c-9e91-2c8be861bb8a",
"currentSizeBytes": 18639885,
"totalSizeBytes": 142639924,
"enabled": true,
"arrowCachingEnabled": false,
"status": {
"config": "OK",
"refresh": "SCHEDULED",
"availability": "AVAILABLE",
"combinedStatus": "CAN_ACCELERATE",
"failureCount": 0,
"lastDataFetch": "2023-01-10T17:12:40.244Z",
"expiresAt": "3022-07-05T19:19:40.244Z"
},
"dimensionFields": [
{
"name": "pickup_date"
},
{
"name": "pickup_datetime",
"granularity": "DATE"
},
{
"name": "dropoff_date"
},
{
"name": "dropoff_datetime",
"granularity": "DATE"
},
{
"name": "passenger_count"
},
{
"name": "total_amount"
},
{
"name": "trip_distance_mi"
}
],
"measureFields": [
{
"name": "passenger_count",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "trip_distance_mi",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "fare_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "surcharge",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "tip_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "total_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
}
],
"distributionFields": [
{
"name": "trip_distance_mi"
},
{
"name": "total_amount"
}
],
"partitionFields": [
{
"name": "dropoff_date"
},
{
"name": "passenger_count"
}
],
"sortFields": [
{
"name": "trip_distance_mi"
}
],
"partitionDistributionStrategy": "CONSOLIDATED",
"canView": true,
"canAlter": true,
"entityType": "reflection"
}

Response Status Codes

200   OK

401   Unauthorized

404   Not Found

500   Internal Server Error

Retrieving All Reflections for a Dataset

Retrieve all raw and aggregation reflections for the specified dataset.

Method and URL
GET /api/v3/dataset/{dataset-id}/reflection

Parameters

dataset-id Path   String (UUID)

Unique identifier of the dataset whose reflections you want to retrieve.

Example: 3cbab7b3-ee82-44c1-abcc-e86d56078d4d

Example Request
curl -X GET 'https://{hostname}/api/v3/dataset/3cbab7b3-ee82-44c1-abcc-e86d56078d4d/reflection'
--header 'Authorization: Bearer <PersonalAccessToken>' \
--header 'Content-Type: application/json'

In the response for a request to retrieve all reflections for a dataset, the reflection objects are wrapped with a data array. Each object in the data array represents one reflection.

Example Response
{
"data": [
{
"id": "23f75eb1-045f-447f-b3fa-374377877569",
"type": "RAW",
"name": "Raw Reflection",
"tag": "K9J2SHE0c+Q=",
"createdAt": "2023-02-03T16:38:27.770Z",
"updatedAt": "2023-02-03T16:38:27.770Z",
"datasetId": "3cbab7b3-ee82-44c1-abcc-e86d56078d4d",
"currentSizeBytes": 0,
"totalSizeBytes": 0,
"enabled": true,
"arrowCachingEnabled": false,
"status": {
"config": "OK",
"refresh": "MANUAL",
"availability": "AVAILABLE",
"combinedStatus": "CAN_ACCELERATE",
"failureCount": 0,
"lastDataFetch": "2023-02-03T16:38:27.780Z",
"expiresAt": "3022-06-06T16:38:27.780Z"
},
"displayFields": [
{
"name": "pickup_datetime"
},
{
"name": "passenger_count"
},
{
"name": "trip_distance_mi"
},
{
"name": "fare_amount"
},
{
"name": "tip_amount"
},
{
"name": "total_amount"
}
],
"partitionDistributionStrategy": "CONSOLIDATED",
"canView": true,
"canAlter": true,
"entityType": "reflection"
},
{
"id": "3cbab7b3-ee82-44c1-abcc-e86d56078d4d",
"type": "AGGREGATION",
"name": "Aggregation Reflection",
"tag": "Mc4hDFk5JR8=",
"createdAt": "2023-02-03T16:39:40.556Z",
"updatedAt": "2023-02-03T16:39:40.556Z",
"datasetId": "1acab7b3-ee82-44c1-abcc-e86d56078d4d",
"currentSizeBytes": 0,
"totalSizeBytes": 0,
"enabled": true,
"arrowCachingEnabled": false,
"status": {
"config": "OK",
"refresh": "MANUAL",
"availability": "AVAILABLE",
"combinedStatus": "CAN_ACCELERATE",
"failureCount": 0,
"lastDataFetch": "2023-02-03T16:39:40.568Z",
"expiresAt": "3022-06-06T16:39:40.568Z"
},
"dimensionFields": [
{
"name": "passenger_count"
},
{
"name": "pickup_datetime",
"granularity": "DATE"
}
],
"measureFields": [
{
"name": "trip_distance_mi",
"measureTypeList": [
"COUNT",
"SUM"
]
},
{
"name": "total_amount",
"measureTypeList": [
"COUNT",
"SUM"
]
},
{
"name": "tip_amount",
"measureTypeList": [
"COUNT",
"SUM"
]
},
{
"name": "fare_amount",
"measureTypeList": [
"COUNT",
"SUM"
]
}
],
"partitionDistributionStrategy": "CONSOLIDATED",
"canView": true,
"canAlter": true,
"entityType": "reflection"
}
],
"canAlterReflections": true
}

Response Status Codes

200   OK

401   Unauthorized

404   Not Found

405   Method Not Allowed

500   Internal Server Error

Updating a Reflection

Update the specified reflection.

Method and URL
PUT /api/v3/reflection/{id}

Parameters

id Path   String (UUID)

Unique identifier of the reflection that you want to update.

Example: 836eae91-306e-487b-a687-31c999653a86


type Body   String

Reflection type. For more information, read Types of Reflections.

Enum: RAW, AGGREGATION

Example: AGGREGATION


name Body   String

Name to use for the reflection.

Example: New Aggregation Reflection


tag Body   String

Unique identifier of the most recent version of the reflection. Dremio uses the tag to ensure that you are updating the most recent version of the reflection.

Example: ZpzGgxw2l04=


datasetId Body   String (UUID)

Unique identifier of the anchor dataset associated with the reflection.

Example: gc870df7-ddf7-4d1e-bb9e-beef28ae773f


enabled Body   Boolean

If the reflection should be available for accelerating queries, set to true. Otherwise, set to false.

Example: false


arrowCachingEnabled Body   Boolean   Optional

If Dremio should convert data from the reflection's Parquet files to Apache Arrow format when copying that data to executor nodes, set to true. Otherwise, set to false (default).

Example: true


displayFields Body   Array of Object

Information about the fields to display from the anchor dataset. The displayfields array must list every field in the anchor dataset or the reflection fails. Each displayFields object contains one attribute: name. Valid only for raw reflections.

Example: [{"name": "pickup_datetime"},{"name": "passenger_count"},{"name": "trip_distance_mi"},{"name": "fare_amount"},{"name": "tip_amount"},{"name": "total_amount"}]


dimensionFields Body   Array of Object

Information about the dimension fields from the anchor dataset to use in the reflection. Dimension fields are the fields you expect to group by when analyzing data. Each dimensionFields object contains two attributes: name and granularity. Valid only for aggregation reflections. If you omit the dimensionFields object in a PUT request, Dremio removes all existing dimension fields from the reflection. To keep existing dimension fields while making other updates, duplicate the existing dimensionFields array in the PUT request.

Example: [{"name": "pickup_datetime","granularity": "DATE"},{"name": "passenger_count","granularity": "DATE"},{"name": "total_amount","granularity": "DATE"},{"name": "trip_distance_mi","granularity": "DATE"}]


measureFields Body   Array of Object

Information about the measure fields from the anchor dataset to use in the reflection. Measure fields are the fields you expect to use for calculations when analyzing the data. Each measureFields object contains two attributes: name and measureTypeList. Valid only for aggregation reflections. If you omit the measureFields object in a PUT request, Dremio removes all existing measure fields from the reflection. To keep existing measure fields while making other updates, duplicate the existing measureFields array in the PUT request.

Example: [{"name": "passenger_count","measureTypeList": ["SUM", "COUNT"]},{"name": "trip_distance_mi","measureTypeList": ["SUM", "COUNT"]},{"name": "fare_amount","measureTypeList": ["SUM", "COUNT"]},{"name": "tip_amount","measureTypeList": ["SUM", "COUNT"]},{"name": "total_amount","measureTypeList": ["SUM", "COUNT"]}]


distributionFields Body   Array of Object   Optional

Information about the distribution fields from the anchor dataset to use for co-locating and co-partitioning data from multiple datasets across nodes. Each distributionFields object contains one attribute: name.
If you omit the distributionFields object in a PUT request, Dremio removes all existing distribution fields from the reflection. To keep existing distribution fields while making other updates, duplicate the existing distributionFields array in the PUT request.

Example: [{"name": "trip_distance_mi"},{"name": "total_amount"}]


partitionFields Body   Array of Object   Optional

Information about the fields from the anchor dataset to use to partition data in the reflection. Each field name is listed as an individual object. If you omit the partitionFields object in a PUT request, Dremio removes all existing partition fields from the reflection. To keep existing partition fields while making other updates, duplicate the existing partitionFields array in the PUT request. For more information, read Horizontally Partition Reflections that Have Many Rows.

Example: [{"name": "pickup_datetime"},{"name": "passenger_count"}]


sortFields Body   Array of Object

Information about the fields from the anchor dataset to use for sorting in the reflection. Each sortFields object contains one attribute: name. If you omit the sortFields object in a PUT request, Dremio removes all existing sort fields from the reflection. To keep existing sort fields while making other updates, duplicate the existing sortFields array in the PUT request. For more information, read Sort Reflections on High-Cardinality Fields.

Example: "name": "trip_distance_mi"


partitionDistributionStrategy Body   String   Optional

Method to use to optimize data compression when executing reflections. If set to CONSOLIDATED (default), Dremio minimizes the number of files produced. If set to STRIPED, Dremio minimizes the time required to refresh the reflection.

Enum: CONSOLIDATED, STRIPED

Example: CONSOLIDATED

Parameters of the displayFields Object

name Body   String

Name of the field to display from the anchor dataset.

Example: "name": "pickup_datetime"

Parameters of the dimensionFields Object

name Body   String

Name of the field from the anchor dataset to configure as a dimension for the reflection.

Example: "name": "pickup_datetime"


granularity Body   String

Grouping to use for the dimension field. If Dremio should automatically extract the day-level date value and use it as the grouping value in the reflection, DATE. If Dremio should use the original value for grouping, NORMAL.

Enum: DATE, NORMAL

Example: "granularity": "DATE"

Parameters of the measureFields Object

name Body   String

Name of the field from the anchor dataset that you expect to use in calculations. Fields of types LIST, MAP, and UNION are not valid measureFields.

Example: "name": "passenger_count"


measureTypeList Body   Array of String

Name of the field from the anchor dataset to use for co-locating and co-partitioning data from multiple datasets across nodes. Every field listed as a distribution field must also be listed as a dimension field.

Enum: APPROX_COUNT_DISTINCT, MIN, MAX, UNKNOWN, SUM, COUNT

Example: ["SUM", "COUNT"]

Parameters of the distributionFields Object

name Body   String

Name of the field from the anchor dataset to use for co-locating and co-partitioning data from multiple datasets across nodes. In aggregation reflections, every field listed as a distribution field must also be listed as a dimension field.

Example: "name": "pickup_datetime"

Parameters of the partitionFields Object

name Body   String

Name of the field from the anchor dataset on which the rows in the reflection are to be partitioned. If a column is listed as a partition column, it cannot also be listed as a sort column for the same reflection. In aggregation reflections, each column specified as a partition column or used in transform must also be listed as a dimension column. In raw reflections, each column specified as a partition column or used in transform must also be listed as a display column.

Example: "name": "dropoff_date"


transform Object

The type of partition transform that is applied. The value is an enum. The types are:

IDENTITY: Creates one partition per value. This is the default transform. If no transform is specified for a field named by the name property, an IDENTITY transform is performed.

IDENTITY Example
{
"name": "passenger_count",
"transform": {
"type": "IDENTITY"
}
}

YEAR: Partitions by year. The field must use the TIMESTAMP or DATE data type.

YEAR Example
{
"name": "pickup_datetime",
"transform": {
"type": "YEAR"
}
}

MONTH: Partitions by month. The field must use the TIMESTAMP or DATE data type.

MONTH Example
{
"name": "pickup_datetime",
"transform": {
"type": "MONTH"
}
}

DAY: Partitions on the equivalent of dateint. The field must use the TIMESTAMP or DATE data type.

DAY Example
{
"name": "pickup_datetime",
"transform": {
"type": "DAY"
}
}

HOUR: Partitions on the equivalent of dateint and hour. The field must use the TIMESTAMP data type.

HOUR Example
{
"name": "pickup_datetime",
"transform": {
"type": "HOUR"
}
}

BUCKET: Partitions data into the number of partitions specified by an integer. For example, if the integer value N is specified, the data is partitioned into N, or (0 to (N-1)), partitions. The partition in which an individual row is stored is determined by hashing the field value and then calculating <hash_value> mod N. If the result is 0, the row is placed in partition 0; if the result is 1, the row is placed in partition 1; and so on.

This value must be followed by a bucketTransform object. This object takes one property: bucketCount. This property takes an integer value.

BUCKET Example
{
"name": "pickup_datetime",
"transform": {
"type": "BUCKET",
"bucketTransform": {
"bucketCount": 1000
}
}
}

TRUNCATE: If the specified field uses the string data type, truncates strings to a maximum of the number of characters specified by an integer. For example, suppose the specified transform is truncate(1, stateUS). A value of CA is truncated to C, and the row is placed in partition C. A value of CO is also truncated to C, and the row is also placed in partition C.

If the specified field uses the integer or long data type, truncates field values in the following way: For any truncate(L, col), truncates the field value to the biggest multiple of L that is smaller than the field value. For example, suppose the specified transform is truncate(10, intField). A value of 1 is truncated to 0 and the row is placed in the partition 0. A value of 247 is truncated to 240 and the row is placed in partition 240. If the transform is truncate(3, intField), a value of 13 is truncated to 12 and the row is placed in partition 12. A value of 255 is not truncated, because it is divisble by 3, and the row is placed in partition 255. This value must be followed by a truncateTransform object.

This object takes one property: truncateLength. This property takes an integer value.

note

The truncate transform does not change field values. It uses field values to calculate the correct partitions in which to place rows.

TRUNCATE Example
{ 
"name": "pickup_hour",
"transform": {
"type": "TRUNCATE",
"truncateTransform": {
"truncateLength": 3
}
}
}

Parameters of the sortFields Object

name Body   String

Name of the field from the anchor dataset to use for sorting in the reflection. Every field listed as a sort field must also be listed as a dimension field. If you list a field as a sort field, you cannot list the same field as a partition field in the same reflection.

Example: "name": "pickup_datetime"

Example Request
curl -X PUT 'https://{hostname}/api/v3/reflection/836eae91-306e-487b-a687-31c999653a86' \
--header 'Authorization: Bearer <PersonalAccessToken>' \
--header 'Content-Type: application/json' \
--data-raw '{
"id": "836eae91-306e-487b-a687-31c999653a86",
"type": "AGGREGATION",
"name": "New Aggregation Reflection",
"tag": "sEHieiuinqE=",
"datasetId": "gc870df7-ddf7-4d1e-bb9e-beef28ae773f",
"enabled": false,
"arrowCachingEnabled": true,
"dimensionFields": [
{
"name": "pickup_datetime",
"granularity": "DATE"
},
{
"name": "passenger_count"
},
{
"name": "total_amount"
},
{
"name": "trip_distance_mi"
}
],
"measureFields": [
{
"name": "passenger_count",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "trip_distance_mi",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "fare_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "tip_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "total_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
}
],
"distributionFields": [
{
"name": "trip_distance_mi"
},
{
"name": "total_amount"
}
],
"partitionFields": [
{
"name": "pickup_datetime"
},
{
"name": "passenger_count"
}
],
"sortFields": [
{
"name": "trip_distance_mi"
}
],
"entityType": "reflection"
}'
Example Response
{
"id": "836eae91-306e-487b-a687-31c999653a86",
"type": "AGGREGATION",
"name": "New Aggregation Reflection",
"tag": "nRPbilwodqC=",
"createdAt": "2023-01-30T14:35:19.192Z",
"updatedAt": "2023-01-30T14:35:19.192Z",
"datasetId": "gc870df7-ddf7-4d1e-bb9e-beef28ae773f",
"currentSizeBytes": 0,
"totalSizeBytes": 0,
"enabled": false,
"arrowCachingEnabled": true,
"status": {
"config": "OK",
"refresh": "SCHEDULED",
"availability": "NONE",
"combinedStatus": "DISABLED",
"failureCount": 0,
"lastDataFetch": "1969-12-31T23:59:59.999Z",
"expiresAt": "1969-12-31T23:59:59.999Z"
},
"dimensionFields": [
{
"name": "pickup_datetime",
"granularity": "DATE"
},
{
"name": "passenger_count"
},
{
"name": "total_amount"
},
{
"name": "trip_distance_mi"
}
],
"measureFields": [
{
"name": "passenger_count",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "trip_distance_mi",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "fare_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "tip_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
},
{
"name": "total_amount",
"measureTypeList": [
"SUM",
"COUNT"
]
}
],
"distributionFields": [
{
"name": "trip_distance_mi"
},
{
"name": "total_amount"
}
],
"partitionFields": [
{
"name": "pickup_datetime"
},
{
"name": "passenger_count"
}
],
"sortFields": [
{
"name": "trip_distance_mi"
}
],
"partitionDistributionStrategy": "CONSOLIDATED",
"canView": true,
"canAlter": true,
"entityType": "reflection"
}

Response Status Codes

200   OK

401   Unauthorized

404   Not Found

409   Conflict

500   Internal Server Error

Deleting a Reflection

Delete the specified reflection.

Method and URL
DELETE /api/v3/reflection/{id}

Parameters

id Path   String (UUID)

Unique identifier of the reflection that you want to delete.

Example: 95dda9dd-2371-467f-b68d-fc4c5ea57a8b

Example Request
curl -X DELETE 'https://{hostname}/api/v3/reflection/95dda9dd-2371-467f-b68d-fc4c5ea57a8b'
--header 'Authorization: Bearer <PersonalAccessToken>' \
--header 'Content-Type:application/json'
Example Response
No response

Response Status Codes

200   OK

401   Unauthorized

404   Not Found

405   Method Not Allowed