Skip to main content

Autonomous Reflections

Autonomous Reflections observe your actual query traffic and automatically create and manage Reflections to accelerate frequent and expensive queries. They are designed to complement Manual Reflections, not replace them. You cover the access patterns you know when building a use case; Autonomous Reflections cover the patterns you don't.

For an overview of how Reflections work, see How Reflections Work.

note

Autonomous Reflections support Iceberg tables, Parquet datasets, and any views built on these datasets. For other data formats, use Manual Reflections.

How Workloads Are Autonomously Accelerated​

Dremio continuously collects metadata from user queries. The autonomous engine runs daily at midnight UTC, analyzing query patterns from the last 7 days and creating Reflections that accelerate frequent and expensive queries. This covers queries on existing views, queries with joins written directly on base tables, and analytical queries that summarize data, such as those submitted by AI agents and BI dashboards.

Query Qualification​

Only queries meeting the following criteria are considered:

  1. Based on Iceberg tables, Parquet datasets, or views built on them. Queries referencing other dataset types, either directly or via a view, are excluded.
  2. Execution time longer than one second.

Dremio may create system-managed views to anchor raw or aggregation Reflections. These views cannot be modified or referenced by users. Admins can drop them, which also deletes the associated Reflection.

Reflection Limits​

Dremio can create up to 100 Reflections total, with a maximum of 10 new Reflections created per day. The actual number depends on query patterns.

How Autonomous Reflections Are Maintained​

Autonomous Reflections use the refresh policy configured for the underlying data source or table, with table-level policies taking precedence over source-level policies. The refresh behavior depends on the policy in effect:

  • Live refresh: Refreshed when the table is modified via Dremio (triggered immediately) or other engines (Dremio polls tables approximately every 20 seconds to detect changes).
  • Period: Refreshed at the configured interval (hours, days, or weeks).
  • Schedule: Refreshed at the configured time on the specified days of the week.

To ensure Autonomous Reflections on Iceberg tables refresh whenever source data changes, set the refresh policy on the data source or table to Live refresh. See Specify the Reflection Refresh Policy. For Parquet datasets, use a Period or Schedule policy aligned with how often your source data changes.

Dremio periodically runs OPTIMIZE TABLE and VACUUM TABLE to keep materializations consistently performant. For Reflections with aggregations that are incrementally refreshed, Dremio rolls up smaller inserts to further improve query performance.

Refresh engine: When a project is created, Dremio automatically provisions a small internal refresh engine dedicated to executing Autonomous Reflection refresh jobs. The engine shuts down after 30 seconds of idle time to minimize resource usage and cost.

Data Freshness​

For Reflections backed by Iceberg tables, Dremio verifies freshness at query-planning time and excludes any Reflection that is not current, falling back to the raw source. For Parquet datasets, freshness depends on the configured refresh policy. Dremio has no metadata to interrogate at plan time, so keeping refresh schedules aligned with how frequently the source data changes is important.

Monitor Reflections​

To view Autonomous Reflections created for your project and their metadata (including status, score, footprint, and queries accelerated), see View Reflection Details.

To view the history of changes to Autonomous Reflections in the last 30 days:

  1. Go to Admin > Project > Reflections.
  2. Click History Log.

Remove Reflections​

Autonomous Reflections can be removed in two ways:

  1. Automatic removal: When a Reflection's score falls below the threshold, it is disabled for 7 days before being automatically dropped. Admins can view disabled Autonomous Reflections in the history log.
  2. Manual removal: Admins can manually drop Autonomous Reflections at any time. If an admin manually drops the same Autonomous Reflection three times, Dremio will not recreate it for 90 days.

Disable Autonomous Reflections​

Every project created in Dremio is automatically accelerated with Autonomous Reflections. To disable them for a project:

  1. Go to Admin > Project > Preferences.
  2. Toggle the Autonomous Reflections setting to off.