Refresh Reflections

You can configure refresh settings for all Reflections, whether created manually or managed by Autonomous Reflections. The refresh policy is configured on the underlying source or table, with table-level policies taking precedence over source-level policies.

Types of Reflection Refresh

Dremio supports two types of refresh. It uses incremental refresh wherever possible, falling back to a full refresh when conditions require it.

• Incremental refresh: Only the data that has changed since the last refresh is read and merged into the existing materialization.
• Full refresh: A new materialization is built from scratch and supersedes the existing one once complete.

Incremental Refreshes

Dremio uses incremental refresh wherever possible, falling back to a full refresh when incremental is not applicable. The initial refresh of any Reflection is always a full refresh.

Iceberg and Parquet Tables

Two types of incremental refresh are supported for Iceberg tables and Parquet datasets:

• Iceberg snapshot-based (append-only changes): Dremio reads only the Iceberg snapshots added since the last refresh. OPTIMIZE TABLE operations on Iceberg tables are compatible with this type of refresh.
• Partition-based (non-append changes): Used when changes include updates or deletes (UPDATE, DELETE, MERGE) on Iceberg tables, or DML-equivalent operations on AWS Glue Parquet datasets. Dremio identifies and reprocesses only the affected partitions. Both the anchor table and the Reflection must be partitioned with compatible partition transforms.

All Other Tables

For non-Iceberg/Parquet tables, incremental refresh is append-only.

Dremio identifies new records using one of two methods:

• Folder-based datasets (JSON, Text): Dremio automatically detects new files added to the folder since the last refresh. Changes to existing files are not detected automatically. To force Dremio to re-read modified files, run ALTER TABLE ... REFRESH METADATA FORCE UPDATE.
• Databases: An administrator specifies a strictly monotonically increasing field, such as an auto-incrementing key, of type BigInt, Int, Timestamp, Date, Varchar, Float, Double, or Decimal. Dremio uses this field to fetch records created since the last refresh.

Full Refreshes

When any of the following conditions apply, Dremio performs a full refresh:

• First refresh of any Reflection
• Reflection is defined on a view using nested GROUP BY
• Reflection is defined on a view using UNION (distinct) combined with aggregation functions — UNION ALL and UNION (distinct) without aggregation functions do not force a full refresh
• Reflection is defined on a view using window functions
• Reflection is defined on a view using unsupported join types — INNER JOIN, CROSS JOIN, and outer joins with null-rejecting filters are supported
• More than one source table has changed since the last refresh
• A source table that changed appears more than once in the view
• Anchor table and Reflection have incompatible partition schemes (Iceberg and Parquet, non-append changes only — keep partition schemes in sync via the Advanced editor or ALTER TABLE)

SQL constructs not listed here, such as filters and projections, do not affect refresh eligibility.

For non-Iceberg/Parquet tables, the following additional conditions also force a full refresh:

• Reflection is defined on a view using any join
• Reflection is defined on a table promoted from a file rather than a folder, or a view based on such a table

For information about partitioning Reflections and applying partition transforms, see Horizontally Partition Reflections that Have Many Rows and Partition Reflections to Allow for Partition-Based Incremental Refreshes.

Specify the Reflection Policies

Source settings control the refresh policy for all Reflections on that source's tables, with table-level policies overriding source-level settings.

To configure refresh settings:

1. In the Dremio console, click in the side navigation bar to open the Open Catalog page.

2. Navigate to a source or a table:

   • Sources: On the Open Catalog page, under Connections, right-click the source and select Settings.
   • Tables: On the Open Catalog page, locate the table, hover over its row, and click to the right.

3. Select Reflection refresh in the source/table settings sidebar.

Refresh Policy

Select a policy type:

Refresh policy type	Description
Never refresh	Reflections are not refreshed.
Refresh every (default)	Reflections refresh at the specified number of hours, days, or weeks. The default is one hour.
Set refresh schedule	Reflections refresh at a specific time on the specified days of the week, in UTC. The default is every day at 8:00 a.m. (UTC).
Live refresh (table-level only)	Reflections automatically refresh when underlying Iceberg tables are updated. They are also updated based on the minimum refresh frequency defined by the source-level policy. Only available for Iceberg tables.

Click Save. Your changes go into effect immediately.

Expiration Policy

The expiration policy controls how long a materialization remains eligible to accelerate queries after its last refresh. Setting the refresh interval shorter than the expiration window ensures no gap in acceleration. If a refresh fails, the existing materialization continues to serve queries until the window closes.

For Reflections using Live Refresh, the table-level expiration setting does not apply. Instead, the source-level expiry policy serves as a backstop: the materialization will expire if the table goes unmodified beyond that window. Because Live Refresh tables are expected to see frequent updates, this backstop is unlikely to take effect in practice. For all other Reflection types, you can set a custom expiration window or disable expiration entirely.

By default, materializations expire after 3 hours. With the default refresh period of 1 hour, this gives a 2-hour buffer before a stale materialization expires. Never expire is not enabled by default and must be explicitly checked to disable expiration.

Enter a duration, or check Never expire to disable expiration entirely.

Click Save. The changes take effect on the next refresh.

User-Managed Refresh Strategy

By default, Reflections use the DREMIO_MANAGED source consistency, which means Dremio automatically refreshes Reflections based on configured source and table refresh policies and data change detection. When all Reflections are set to DREMIO_MANAGED, they eventually become consistent with their source data. Over time, all Reflections will reflect the same state of the source tables.

You can use USER_MANAGED source consistency to take precise control over which Reflections refresh and when. This is useful when:

• You only want to refresh Reflections that are actively queried, rather than keeping all Reflections refreshed at the same time.
• You need precise control over when Reflections refresh, independent of upstream Reflection refresh cycles.
• The cost of keeping all Reflections current outweighs the benefit.

For example, if Reflection A joins Reflection B on RDBMS data and Reflection C on Iceberg data, you can refresh A and C to pick up intraday Iceberg changes without triggering a refresh of B.

Key characteristics of USER_MANAGED Reflections include:

• No automatic refresh: The Reflection does not update on a schedule or in response to data changes. If another Reflection accelerates a USER_MANAGED Reflection, updates to that upstream Reflection do not trigger the USER_MANAGED Reflection to refresh.

• Explicit refresh only: The Reflection refreshes only when you explicitly trigger a refresh using ALTER TABLE, ALTER VIEW, or the Reflection API.

• Isolated refresh: Only the targeted Reflection is refreshed. Upstream and downstream Reflections are not automatically triggered. You are responsible for refreshing upstream reflections if needed.

important

Reflections with different source consistency will not mix. As a result, a Dremio-managed reflection cannot be created with acceleration from a user-managed reflection and vice versa. Otherwise, data with different source consistencies will mix. In either case, this could compromise the Reflections' intended data freshness.

Set Source Consistency

You can set the source consistency when using the Reflection API, ALTER TABLE, or ALTER VIEW to create a Reflection. See the example below.

Set the source consistency when creating a Reflection with SQL

ALTER TABLE mySchema.myTable
  CREATE RAW REFLECTION myReflection
    USING DISPLAY (col1, col2)
    SOURCE CONSISTENCY USER MANAGED;

View the Reflection Refresh History

You can view the history of refresh jobs run for any Reflection.

To view the refresh history:

1. In the Dremio console, go to the catalog or folder that lists the table or view from which the Reflection was created.
2. Hover over the row for the table or view and click on the right.
3. Select Reflections in the dataset settings sidebar.
4. Click History in the heading for the Reflection.

The Jobs page is opened with the ID of the Reflection in the search box, and only jobs related to that ID are listed.

When a Reflection is refreshed, Dremio runs a single job with two steps:

• The first step writes the query results as a materialization to the distributed acceleration storage by running a REFRESH REFLECTION command.
• The second step registers the materialization table and its metadata with the catalog so that the query optimizer can find the Reflection's definition and structure.

To find out which type of refresh was performed:

1. Click the ID of the job that ran the REFRESH REFLECTION command.
2. Click the Raw Profile tab.
3. Click the Planning tab.
4. Scroll down to the Refresh Decision section.

Retry Reflection Refreshes

When a Reflection refresh job fails, Dremio retries the refresh according to a uniform policy. This policy is designed to balance resource consumption with the need to keep Reflection data up to date. It prioritizes newly failed Reflections to reduce excessive retries on persistent failures and helps ensure that Reflection data does not become overly stale.

After a refresh failure, Dremio retries automatically with increasing gaps between each attempt, up to a maximum interval of 4 hours. Dremio stops retrying after 24 attempts or when the 72-hour retry window is reached, whichever comes first.

Trigger Reflection Refreshes

You can click a button to start the refresh of all of the Reflections that are defined on a table or on views derived from that table.

To trigger a refresh manually:

1. Locate the table.
2. Hover over the row in which it appears and click to the right.
3. In the sidebar of the Dataset Settings window, click Reflection Refresh.
4. Click Refresh Now. The message "All dependent Reflections will be refreshed." appears at the top of the screen.
5. Click Save.

You can refresh Reflections by using the Reflection API, the Catalog API, and the SQL commands ALTER TABLE and ALTER VIEW.

• With the Reflection API, you specify the ID or name of a Reflection. See Refresh a Reflection.
• With the Catalog API, you specify the ID of a table or view that the Reflections are defined on. See Refresh the Reflections on a Table and Refresh the Reflections on a View.
• With the ALTER TABLE and ALTER VIEW commands, you specify the path and name of the table or view that the Reflections are defined on.

Refresh Cascade

Regardless of which method is used, the refresh cascades as follows. Refreshes are ordered bottom-up: a Reflection on a view cannot refresh until all Reflections on its upstream tables and views have completed.

• If the target is a view, Reflections on all upstream tables and views the anchor view is built on are also refreshed.
• If the target is a table, Reflections on all views built on that table are also refreshed.

For example, suppose that you had the following tables and views, with Reflections R1 through R5 defined on them:

         View2(R5)
         /       \
     View1(R3) Table3(R4)
    /       \
Table1(R1) Table2(R2)

• Refreshing Reflection R5 also refreshes R1, R2, R3, and R4.
• Refreshing Reflection R4 also refreshes R5.
• Refreshing Reflection R3 also refreshes R1, R2, and R5.
• Refreshing Reflection R2 also refreshes R3 and R5.
• Refreshing Reflection R1 also refreshes R3 and R5.