Skip to main content

Refresh Reflections

Refresh settings apply to all Reflections, whether created manually or managed by Autonomous Reflections. Autonomous Reflections use the refresh policy configured on the underlying data source or table, with table-level policies taking precedence over source-level policies.

When a Reflection's data becomes stale, refreshing it triggers two updates:

  • The data stored in the Apache Iceberg table for the Reflection is updated.
  • The metadata that stores details about the Reflection is updated.
note

Dremio does not refresh the data that external Reflections are mapped to.

Types of Reflection Refresh​

How Reflections are refreshed depend on the format of the base table.

Apache Iceberg Tables and Parquet Datasets​

Common catalogs include:

  • Dremio Open Catalog serves data as standard Iceberg via the Iceberg REST catalog protocol.
  • Iceberg REST catalogs, such as Snowflake's Iceberg catalog, follow the same path.
  • AWS Glue provides a consistent access layer for Iceberg and Parquet tables.
  • Unity Catalog exposes tables to Dremio as Iceberg, regardless of the underlying storage format. For Delta Lake data, Unity Catalog is the recommended path.

Dremio can also read Iceberg and Parquet datasets directly from object storage sources such as Amazon S3 and Azure Storage, without a catalog layer.

Incremental Refreshes​

There are two types of incremental refreshes, depending on whether the changes to the anchor table are append-only or include updates and deletes. The initial refresh of a Reflection is always a full refresh.

Incremental Refreshes: Append-Only Changes​

note

Optimize operations on Iceberg tables are also supported for this type of incremental refresh.

This type of incremental refresh is used only when the changes to the anchor table are appends and do not include updates or deletes. There are two cases to consider:

  • When a Reflection is defined on one anchor table

    For a single anchor table, the refresh reads only the changes between the current snapshot and the snapshot at the time of the last refresh.

  • When a Reflection is defined on a view that joins two or more anchor tables

    Whether an incremental refresh can be performed depends on how many anchor tables have changed since the last refresh:

    • One table has changed: an incremental refresh is performed, reading only the delta rows from the changed table. This is safe because the unchanged tables remain stable. The new rows can be joined to the existing data and merged into the materialization without recomputing the whole result. If the Reflection includes an aggregation (such as a GROUP BY across the join), the incremental rows are aggregated separately and merged into the stored aggregate rather than requiring a full recompute.
    • Two or more tables have changed: a full refresh is performed. When rows on both sides of a join may have shifted, there is no safe way to compute a partial incremental result, so Dremio recomputes from scratch.

Incremental Refreshes: Non-Append Changes​

For Iceberg tables, this type of incremental refresh is used when the changes are DML operations that delete or modify the data (UPDATE, DELETE, MERGE).

For AWS Glue sources, non-append operations include DML-equivalent operations performed on Parquet datasets.

Both the anchor table and the Reflection must be partitioned, and the partition transforms that they use must be compatible.

There are two cases to consider:

  • When a Reflection is defined on one anchor table

    For a single anchor table, Dremio uses Iceberg metadata to identify which partitions changed and refreshes only those partitions.

  • When a Reflection is defined on a view that joins two or more anchor tables

    The same one-table/many-tables rule applies. If one anchor has changed, Dremio uses Iceberg metadata to identify and refresh only the affected partitions. If more than one has changed, a full refresh is performed.

For information about partitioning Reflections and applying partition transforms, see the section Horizontally Partition Reflections that Have Many Rows.

For information about partitioning Reflections in ways that are compatible with the partitioning of anchor tables, see Partition Reflections to Allow for Partition-Based Incremental Refreshes.

Full Refreshes​

In a full refresh, a Reflection is dropped, recreated, and loaded.

note

The initial refresh of a Reflection is always a full refresh.

Rules Used to Determine the Refresh Type​

Dremio applies the following rules, in order, to determine which refresh method to use:

  1. The first refresh of any Reflection is always a full refresh.
  2. If the Reflection is defined on a view that uses nested group-bys, unions, window functions, or joins other than inner or cross joins, a full refresh is performed.
  3. If the Reflection is defined on a view that joins two or more anchor tables and more than one has changed since the last refresh, a full refresh is performed.
  4. If the changed anchor table appears multiple times in the view, a full refresh is performed.
  5. If none of the above apply, Dremio uses an incremental refresh. For append-only changes this is snapshot-based. For non-append changes, it is partition-based, provided the anchor table and the Reflection use compatible partition schemes. If the partition schemes are incompatible, Dremio falls back to a full refresh. To avoid this, keep the Reflection's partition scheme in sync with the anchor table's, either in the Advanced view of the Reflection editor or via the ALTER DATASET SQL command.

All Other Tables​

  • Incremental refreshes

    Dremio appends data to the existing data for a Reflection. Incremental refreshes are faster than full refreshes for large Reflections, and are appropriate for Reflections that are defined on tables that are not partitioned.

    There are two ways in which Dremio can identify new records:

    • For directory datasets in S3 or Azure Storage: Dremio can automatically identify new files in the directory that were added after the prior refresh.
    • For all other datasets (such as datasets in relational or NoSQL databases): An administrator specifies a strictly monotonically increasing field, such as an auto-incrementing key, that must be of type BigInt, Int, Timestamp, Date, Varchar, Float, Double, or Decimal. This allows Dremio to find and fetch the records that have been created since the last time the acceleration was incrementally refreshed.
    caution

    Use incremental refreshes only for Reflections that are based on tables and views that are appended to. If records can be updated or deleted in a table or view, use full refreshes for the Reflections that are based on that table or view.

  • Full refreshes

    In a full refresh, the Reflection being refreshed is dropped, recreated, and loaded.

    Full refreshes are always used in these three cases:

    • A Reflection is partitioned on one or more fields.
    • A Reflection is created on a table that was promoted from a file, rather than from a folder, or is created on a view that is based on such a table.
    • A Reflection is created from a view that uses nested group-bys, joins, unions, or window functions.

Specify the Reflection Refresh Policy​

In the settings for a data source, you specify the refresh policy for refreshes of all Reflections that are on the tables in that data source. The default policy is period-based, with one hour between each refresh. If you select a schedule policy, the default is every day at 8:00 a.m. (UTC).

In the settings for a table that is not in the Iceberg or Delta Lake format, you can specify the type of refresh to use for all Reflections that are ultimately derived from the table. The default refresh type is Full refresh.

For tables in all supported table formats, you can specify a refresh policy for Reflection refreshes that overrides the policy specified in the settings for the table's data source. The default policy is the schedule set at the source of the table.

To set the refresh policy on a data source:

  1. In the Dremio console, right-click a data lake or external source.

  2. Select Edit Details.

  3. In the sidebar of the Edit Source window, select Reflection Refresh.

  4. When you are done making your selections, click Save. Your changes go into effect immediately.

To edit the refresh policy on a table:

  1. Locate the table.

  2. Hover over the row in which it appears and click The Settings icon to the right.

  3. Select Reflection Refresh in the dataset settings sidebar.

  4. When you are done making your selections, click Save. Your changes go into effect immediately.

Types of Refresh Policies​

Datasets and sources can set Reflections to refresh according to the following policy types:

Refresh policy typeDescription
NeverReflections are not refreshed.
Period (default)Reflections refresh at the specified number of hours, days, or weeks. The default refresh period is one hour.
ScheduleReflections refresh at a specific time on the specified days of the week, in UTC. The default is every day at 8:00 a.m. (UTC).
Live RefreshReflections automatically refresh when underlying Iceberg tables are updated. They are also updated based on the minimum refresh frequency defined by the source-level policy. This refresh policy is only available for data sources that support the Iceberg table format.

Set the Reflection Expiration Policy​

Rather than delete a Reflection manually, you can specify how long you want Dremio to retain the Reflection before deleting it automatically.

note

Dremio does not allow expiration policies to be set on external Reflections or Reflections using the Live refresh policy.

To set the expiration policy for all Reflections derived from tables in a data source:

  1. Right-click a data lake or external source.

  2. Select Edit Details.

  3. Select Reflection Refresh in the edit source sidebar.

  4. After making your changes, click Save. The changes take effect on the next refresh.

To set the expiration policy on Reflections derived from a particular table:

note

The table must be based on more than one file.

  1. Locate a table.

  2. Click the The Settings icon to its right.

  3. Select Reflection Refresh in the dataset settings sidebar.

  4. After making your changes, click Save. The changes take effect on the next refresh.

User-Managed Refresh Strategy​

By default, Reflections use the DREMIO_MANAGED source consistency, which means Dremio automatically refreshes Reflections based on configured source and table refresh policies and data change detection. When all Reflections are set to DREMIO_MANAGED, they eventually become consistent with their source data. Over time, all Reflections will reflect the same state of the source tables.

You can use USER_MANAGED source consistency to take precise control over which Reflections refresh and when. This is useful when:

  • You only want to refresh Reflections that are actively queried, rather than keeping all Reflections refreshed at the same time.
  • You need precise control over when Reflections refresh, independent of upstream Reflection refresh cycles.
  • The cost of keeping all Reflections current outweighs the benefit.

For example, if Reflection A joins Reflection B on RDBMS data and Reflection C on Iceberg data, you can refresh A and C to pick up intraday Iceberg changes without triggering a refresh of B.

Key characteristics of USER_MANAGED Reflections include:

  • No automatic refresh: The Reflection does not update on a schedule or in response to data changes. If another Reflection accelerates a USER_MANAGED Reflection, updates to that upstream Reflection do not trigger the USER_MANAGED Reflection to refresh.

  • Explicit refresh only: The Reflection refreshes only when you explicitly trigger a refresh using ALTER TABLE, ALTER VIEW, or the Reflection API.

  • Isolated refresh: Only the targeted Reflection is refreshed. Upstream and downstream Reflections are not automatically triggered. You are responsible for refreshing upstream reflections if needed.

important

Reflections with different source consistency will not mix. As a result, a Dremio-managed reflection cannot be created with acceleration from a user-managed reflection and vice versa. Otherwise, data with different source consistencies will mix. In either case, this could compromise the Reflections' intended data freshness.

Set Source Consistency​

You can set the source consistency when using the Reflection API, ALTER TABLE, or ALTER VIEW to create a Reflection. See the example below.

Set the source consistency when creating a Reflection with SQL 
ALTER TABLE mySchema.myTable
  CREATE RAW REFLECTION myReflection
    USING DISPLAY (col1, col2)
    SOURCE CONSISTENCY USER MANAGED;

View the Reflection Refresh History​

You can find out whether a refresh job for a Reflection has run, and how many times refresh jobs for a Reflection have been run.

To view the refresh history:

  1. In the Dremio console, go to the catalog or folder that lists the table or view from which the Reflection was created.

  2. Hover over the row for the table or view.

  3. In the Actions field, click The Settings icon.

  4. Select Reflections in the dataset settings sidebar.

  5. Click History in the heading for the Reflection.

The Jobs page is opened with the ID of the Reflection in the search box, and only jobs related to that ID are listed.

When a Reflection is refreshed, Dremio runs a single job with two steps:

  • The first step writes the query results as a materialization to the distributed acceleration storage by running a REFRESH REFLECTION command.
  • The second step registers the materialization table and its metadata with the catalog so that the query optimizer can find the Reflection's definition and structure.

To find out which type of refresh was performed:

  1. Click the ID of the job that ran the REFRESH REFLECTION command.
  2. Click the Raw Profile tab.
  3. Click the Planning tab.
  4. Scroll down to the Refresh Decision section.

Retry a Reflection Refresh Policy​

When a Reflection refresh job fails, Dremio retries the refresh according to a uniform policy. This policy is designed to balance resource consumption with the need to keep Reflection data up to date. It prioritizes newly failed Reflections to reduce excessive retries on persistent failures and helps ensure that Reflection data does not become overly stale.

After a refresh failure, Dremio's default is to repeat the refresh attempt at exponential intervals up to 4 hours: 1 minute, 2 minutes, 5 minutes, 15 minutes, 30 minutes, 1 hour, 2 hours, and 4 hours. Then, Dremio continues trying to refresh the Reflection every 4 hours.

There are two optimizations for special cases:

  • Long-running refresh jobs: The backoff interval will never be shorter than the last successful duration.

  • Small maximum retry attempts: At least one 4-hour backoff attempt is guaranteed to ensure meaningful coverage of the retry policy.

Dremio stops retrying after 24 attempts, which typically takes about 71 hours and 52 minutes, or when the 72-hour retry window is reached, whichever comes first.

Trigger Reflection Refreshes​

You can click a button to start the refresh of all of the Reflections that are defined on a table or on views derived from that table.

To trigger a refresh manually:

  1. Locate the table.
  2. Hover over the row in which it appears and click The Settings icon to the right.
  3. In the sidebar of the Dataset Settings window, click Reflection Refresh.
  4. Click Refresh Now. The message "All dependent Reflections will be refreshed." appears at the top of the screen.
  5. Click Save.

You can refresh Reflections by using the Reflection API, the Catalog API, and the SQL commands ALTER TABLE and ALTER VIEW.

The refresh action follows this logic for the Reflection API:

  • If the Reflection is defined on a view, the action refreshes all Reflections that are defined on the tables and on downstream/dependent views that the anchor view is itself defined on.

  • If the Reflection is defined on a table, the action refreshes the Reflections that are defined on the table and all Reflections that are defined on the downstream/dependent views of the anchor table.

The refresh action follows similar logic for the Catalog API and the SQL commands:

  • If the action is started on a view, it refreshes all Reflections that are defined on the tables and on downstream/dependent views that the view is itself defined on.

  • If the action is started on a table, it refreshes the Reflections that are defined on the table and all Reflections that are defined on the downstream/dependent views of the anchor table.

For example, suppose that you had the following tables and views, with Reflections R1 through R5 defined on them:

         View2(R5)
         /       \
     View1(R3) Table3(R4)
    /       \
Table1(R1) Table2(R2)
  • Refreshing Reflection R5 through the API also refreshes R1, R2, R3, and R4.
  • Refreshing Reflection R4 through the API also refreshes R5.
  • Refreshing Reflection R3 through the API also refreshes R1, R2, and R5.
  • Refreshing Reflection R2 through the API also refreshes R3 and R5.
  • Refreshing Reflection R1 through the API also refreshes R3 and R5.