Changelog
This changelog provides a detailed record of the previous 12 months of updates and enhancements we have made to improve your Dremio Cloud experience.
June 4, 2025
Updates to Autonomous Reflections are now shown in the history.autonomous_reflection system table.
Updated the following library to address potential security issues:
- Hadoop to Protobuf 3.15.0+ [CVE-2022-3171, CVE-2022-3509, CVE-2022-3510, CVE-2021-22570].
DX-102976
The Reflection refresh retry policy is now enforced within a 72-hour window.
Queries now scan only the required fields from complex structure type columns to prevent errors in query execution.
Fixed an issue where large case expressions were causing the planner to hang.
Fixed a memory leak on the coordinator when queries using Arrow Flight are canceled.
Fixed an issue where queries against tables with equality deletes could fail if the fields used in the deletes had capital letters.
Fixed an issue where Delta tables with unsupported minReaderVersion values were allowed to be promoted, leading to query failures.
Fixed an issue with CREATE TABLE AS SELECT statements on Parquet datasets that could result in an error.
Fixed an issue where partition recommendations were not generated for Reflections defined on views.
The following Iceberg table properties are now supported when configuring Parquet bloom filters:
write.parquet.bloom-filter-enabled.column.<col>enables bloom filters for specific columns.write.parquet.bloom-filter-max-bytessets the maximum size of the bloom filter bitset (default: 1MB).write.parquet.bloom-filter-fpp.column.<col>configures the false positive probability per column (default: 0.01).DX-100948
Fixed an issue where semantic search results did not include parent folders or source paths for entities the user had permission to read.
Queries with too many expressions in WHERE clauses or CASE statements now return a clear error message instead of failing unexpectedly.
Removed the measure_columns column from the sys.reflections table.
Running a SELECT COUNT(*) query on Iceberg tables now leverages metadata to return row counts without scanning the entire table.
Fixed an issue related to CREATE TABLE AS SELECT statements for Parquet datasets with the error message "Multiple entries with same key: $file=$file: Utf8 and $file=$file: Utf8".
Added an INCLUDE option to VACUUM CATALOG to allow targeting specific tables for vacuuming.
You can now manually refresh failed Reflections by clicking the "Refresh now" icon on the Reflections page.
May 7, 2025
Quickly discover your data using the AI-enabled semantic search. Object names, wikis, labels, column names, and other metadata are used to serve the most relevant datasets and entities.
Autonomous Reflections automatically creates and manages reflections, accelerating query performance and optimizing data accessibility. This enables you to derive faster insights, scale effortlessly, and make data-driven decisions.
Dremio now supports a generic Iceberg REST Catalog as a source. This allows you to connect to Iceberg catalogs over the Iceberg REST API. This source is located within the Lakehouse Catalogs.
Support for vended credentials has been added to Snowflake Open Catalog, Unity Catalog, and Iceberg REST Catalog.
Dremio now supports writes against Iceberg REST sources like Snowflake Open Catalog and the generic Iceberg REST Catalog. Unity Catalog remains read only.
Results cache is now supported to accelerate repeated queries.
Incremental Reflection refresh is now supported for tables in Unity Catalog.
Fixed an issue where sources with connection issues could not be modified or deleted.
Fixed an issue where queries returning more than 1 million rows could run slower with results cache.
Fixed an issue where an OversizedAllocationException could occur for collections of variable-length values due to unnecessary data buffer reallocations.
You can now specify columns for both sorting and partitioning in reflections when the partitioning scheme uses a transformation.
Improved Reflection refresh behavior by fixing inaccurate duration reporting and adjusting retry timing for long-running jobs and limited retry configurations.
Fixed an issue with row count estimates in Delta Lake tables that could lead to suboptimal query plans.
April 8, 2025
Updated the following library to address potential security issues:
- org.apache.parquet:parquet-avro to version 1.15.1 [CVE-2025-30065].
DX-102559
March 31, 2025
Fixed an issue where queries using reflections failed or referenced stale materializations.
Fixed an issue where the LEAD() function returned incorrect values for the last row in a partition, where NULL was expected.
Queries accelerated by the results cache now display a lightning bolt icon and indicate a cache hit on the Job Overview page.
March 17, 2025
Dremio now supports Azure Storage container writes with the Azure Blob File System (ABFSS) driver. Iceberg metadata on Azure now defaults to the abfss:// format instead of wasbs://.
You can now use clustering as a straightforward and effective alternative to partitioning. Clustering simplifies processes and helps ensure fast queries on Apache Iceberg tables.
Dremio now supports SAP HANA as a source for AWS projects.
Azure Data Lake Storage Gen1 is no longer supported following its official retirement by Microsoft in February 2024.
You can now update Sample Source and SampleDB source settings with the necessary permissions.
Added an explicit exception for the unsupported IGNORE NULLS option in window functions.
Queries now scan only the required fields from complex structure type columns to prevent errors in query execution.
Fixed an issue where raw or aggregate reflection recommendations were duplicated when a reflection had no available materializations.
Fixed an issue where reflections could have an empty reflection score.
When writing Parquet files to Iceberg tables:
store.parquet.block-sizeanddremio.iceberg.optimize.target_file_size_mbare now ignored.- The
write.parquet.row-group-size-bytesproperty now controls row group and file size, with a default value of 128 MB instead of 256 MB. write.metadata.metrics.max-inferred-column-defaultsandwrite.metadata.metrics.defaultnow control file-level metrics in Iceberg metadata. Thewrite.metadata.metrics.max-inferred-column-defaultsproperty is now 100 instead of unlimited.- The
write.metadata.metrics.defaultmode is nowtruncate(16)instead offull. - The default value for
write.parquet.page-row-limitis now 20,000 instead of unlimited.DX-99417DX-99441DX-99743
Long project owner names are now displayed correctly.
Query completion is no longer blocked by profile updates. The profile now shows a status during updates.
February 21, 2025
Dremio supports Snowflake Open Catalog as a source.
Dremio supports Unity Catalog as a source.
February 19, 2025
Dremio supports SAP HANA as a source for Azure projects.
Fixed an issue where queries could be stuck in a pending state when executor node events are not correctly propagated to the coordinators, causing coordinators not to take appropriate corrective action.
Enhanced default raw reflection matching to include query tree hashes so that more SQL queries can match without direct view references.
Updated the list of Dremio-supported properties and default values in the Supported Properties of Apache Iceberg Tables.
The Iceberg table property compatibility.snapshot-id-inheritance.enabled is no longer written by default when creating Iceberg tables, although it can be used for Iceberg tables (V1) for backward compatibility reasons.
Updated the following library to address potential security issues:
- AsyncHttpClient to 3.0.1 [CVE-2024-53990] DX-98578
Updated the source categories in Dremio Cloud as follows: Lakehouse Catalogs, Object Storage, and Databases.
Fixed some cases where a reflection did not have score on the Reflections page.
Fixed an issue that could cause a query with a LIMIT clause to not be accelerated due to miscalculated row adjustments.
Fixed an issue with zero milliseconds that could occur when using the CURRENT_TIME function.
February 4, 2025
Added isPrimaryCatalog in the Catalog API, so you can update the Arctic primary source.
January 28, 2025
Fixed an issue where the snapshot ID for a table could stop refreshing and cause a reflection to continuously refresh.
Addressed regional endpoint issues with Amazon S3 and AWS STS for users that have projects deployed in VPCs with private subnets that lack public internet connectivity.
January 23, 2025
Dremio supports Azure Synapse Analytics as a source.
Dremio supports Google BigQuery as a source.
You can now select AWS Graviton instances when creating new engines or editing existing engines.
Fixed an issue that could prevent the retry policy for reflection refreshes from working in all cases.
Sonar cards on the Sonar Projects page now show the full name of the user who created the project instead of their email address, and when a user has been deleted from the system, you will see 
Unavailable. Users assigned to the PUBLIC role who do not have access to fetch user details no longer see a gray empty circle in place of user details.
Fixed an issue where a reflection could have an empty reflection score on the Reflections page.
Fixed an issue where clicking on a script in the search results on the Datasets page does not open that script in the SQL Runner.
The script lookup in the All/Mine tabs on the Scripts panel in the SQL Runner was previously case-sensitive. Now, the script lookup is case insensitive.
Because the Arrow caching feature for reflections has been deprecated, any remaining references to Arrow cache for reflections have been removed, such as in the sys.project.reflections system table and APIs.
You can now exclude specific tables directly in the catalog source when running the VACUUM CATALOG SQL command.
Removed unsupported pushdown operations with VARCHAR data types from PostgreSQL advanced relational pushdown (ARP) connectors.
Fixed an issue in some cases that could prevent runtime filtering if the CONVERT_FROM SQL function was used in the query.
January 15, 2025
Fixed an issue that could prevent the OpenID Connect (OIDC) option from appearing in the dropdown menu as an authentication provider for Dremio Enterprise Edition under Authentication on the Organization Settings page.
January 8, 2025
Fixed an issue that could cause Dremio Enterprise Edition to not propagate properly during the reflection refresh process and cause SQL commands containing REFRESH REFLECTIONS to fail.
December 20, 2024
Fixed an issue with the SQL Runner that could prevent users from being able to expand an Arctic catalog in the Data panel to click 
to add a dataset or drag and drop the data into the SQL editor.
December 17, 2024
Parameterized prepared statements are now supported for Arrow Flight SQL JDBC to prevent SQL injection and enable customers to leverage client tools that support this feature. This feature supports SELECT statements.
Fixed a filter pushdown issue that could cause a query to run slowly or return incorrect results.
Fixed an SSL negotiation issue when connecting to Dremio servers through secure connections.
Updated password change behavior in the Dremio console to more effectively handle UI session termination after password rotation.
Fixed an issue that could cause reading tables from the AWS Glue Data Catalog to be slow.
Fixed an issue where a duplicated table schema could be written to its metadata file.
Fixed an issue where REFLECTION REFRESH jobs could fail for reflections involving joins in the query plan if field-based incremental refresh was configured on the underlying datasets. These reflection refreshes will now succeed using full refreshes.
Fixed a issue that could occur when you request reflection recommendations for a specific job and the query you want to accelerate contains a subquery.
Fixed an issue that could cause the SQL Runner to display the view definition of the last executed preview instead of the saved view definition.
Navigating to a dataset with dots in the name on the History tab of the Datasets page will now work as expected.
The SHOW TBLPROPERTIES SQL command will now return the format version for Iceberg tables.
Fixed an issue for SELECT queries when using LIMIT and OFFSET for a value greater than the maximum value for a signed integer. Now LIMIT and OFFSET cannot exceed the maximum integer value.
Deprecated COMPACTION and LOAD MATERIALIZATION for reflection jobs because they optimize non-Iceberg materialization and Dremio now supports only Iceberg materialization. The sys.project.materializations table now only shows REFRESH REFLECTION jobs. COMPACT and LOAD are no longer considered reserved keywords for SQL queries.
Fixed an issue that would cause a query on the sys.project.history tables using 'SYS' uppercase to fail.
Fixed an issue for SELECT queries when using LIMIT and OFFSET for a value greater than the maximum value for a signed integer. Now LIMIT and OFFSET cannot exceed the maximum integer value.
December 3, 2024
Fixed an issue that could cause a query failure due to an exceeded timeout after an elapsed query runtime limit.
Fixed an issue that could cause users without the VIEW JOB HISTORY privilege or ADMIN role to view jobs executed by other users on the Jobs page.
Fixed an issue that could prevent users from enabling or disabling system-wide acceleration on the Reflections page.
Fixed an issue where CLUSTER and CLUSTERING were accidentally added as reserved keywords. They are no longer treated as reserved.
November 19, 2024
The Arrow Flight SQL JDBC driver now supports the project ID parameter for connecting to non-default projects in Dremio Cloud.
When configuring cloud resources in Dremio Cloud, you can specify separate resource groups for compute resources (such as virtual machine scale sets) and network resources (such as virtual networks). While using a single resource group was previously supported, this enhancement offers improved flexibility for cloud infrastructure organization and strengthens security posture.
Dark Mode is now available in Dremio! You can now choose between light mode, dark mode, or system settings. Try it out by going to Account Settings > Appearance.
In preparation for the upcoming End of Life (EOL) of Amazon Linux 2, Dremio has transitioned the base operating system for its executors from Amazon Linux 2 to Ubuntu LTS. This shift ensures continued support, security updates, and improved compatibility with modern infrastructure and libraries.
Updated the following libraries to address potential security issues:
- Ranger client in Dremio from version 1.1 to 1.2 DX-93529
- Avro from 1.11.3 to 1.11.4 [CVE-2024-47561] DX-96442
Fixed a NullPointerException that could occur during a metadata refresh due to closing a filesystem object already evicted from the cache.
Improved the sync time for reflection recommendations.
Fixed an issue that could prevent async Azure reads due to a time zone issue in locations east of Greenwich Mean Time (GMT).
Fixed an issue that could prevent users from being able to run or preview a query in the SQL Runner after viewing the History tab for the query on the Datasets page.
Navigating to the wiki of a dataset from the SQL Runner will no longer cause (edited) to appear next to the dataset name.
Fixed an issue that could prevent changes to the project storage settings from updating on the Project Settings page until the page was refreshed.
Fixed a "Could not update dag for engine schedule" issue that could occur when trying to save edits to the engine schedule on the Engines page.
October 30, 2024
Running a SELECT COUNT(*) query now uses Iceberg metadata instead of scanning the entire Iceberg table to return the total number of rows in a table.
For AWS accounts, fixed an issue where the Save button is disabled while editing the configuration in the catalog settings.
Fixed an issue that could prevent users from editing project settings for projects created using an AWS cloud.
Fixed a issue where decorrelating a subquery with an EXISTS statement and an empty GROUP BY clause could result in incorrect data.
October 16, 2024
You can now access the Arctic UDFs via API which supports CRUD actions.
Fixed an issue where file handles (and HTTP connections) were left opened after reading JSON commit logs for Delta tables within a AWS Glue Data Catalog.
Fixed an issue that could prevent a user from scrolling through the wiki content in the Details tab on the Datasets page.
Fixed an issue with "Go to Table" functionality on the Datasets page that could cause the table definition to be blank on the Data tab when multiple partitions from the same column are added to an Arctic table.
Dremio will now notify you when a view's metadata is out-of-date due to schema changes in the underlying views or tables. The notification will appear on the Data panel in the SQL Runner and in the Details and Lineage tabs on the Datasets page.
Fixed an issue that could cause query results to appear in a new tab when cached results are loading in the SQL Runner.
Creating a new tab while a script is executing will now cause a confirmation dialog to appear in the SQL Runner.
Fixed an issue that prevented non-admin users from saving a view using the Save as View button in the SQL Runner.
The Start Time filter on the Jobs page no longer updates to Custom after a user selects a start time filter, leaves for a short time, and then comes back to the page.
The Visual Profile tab on the Jobs page will now show the correct error message when a visual profile cannot be generated.
When hovering over the tooltip for a reflection score on the Reflections page, the daily query accelerated value will be rounded to the nearest integer.
Fixed a NullPointerException (NPE) that could cause VACUUM jobs for reflections to fail.
HASH_JOIN will now randomize the distribution if there are nulls being generated by a join condition in order to avoid sending the data to the same thread and eventually reduce the skew.
The following words were incorrectly made reserved keywords: CLUSTER, CLUSTERING.
September 23, 2024
Fixed an issue that could occur when attempting to access datasets in the Data panel in the SQL Runner, resulting in a "Something went wrong" error message.
Fixed an issue that could cause views to not save properly for non-admin users when clicking the Save as View button in the SQL Runner.
September 20, 2024
In Enterprise edition, members of the admin role can now configure an OpenID Connect (OIDC) identity provider for authentication under Organization Settings on the Authentication page or using the Identity Providers API. This new authentication method allows organizations to configure SSO with OIDC-compliant identity providers.
You can now connect to Vertica as a source in Dremio.
Azure regions East US 2 and West US 2 have been added for Dremio Cloud.
Create user-defined functions (UDFs) to extend the native capabilities of Dremio’s SQL dialect and reuse business logic across queries. Because UDFs are native, first-class entities in the Arctic catalog, you can seamlessly experiment on and change UDFs using Arctic's branching capabilities.
New SQL commands have been added for UDFs: CREATE FUNCTION, DROP FUNCTION, DESCRIBE FUNCTION, and SHOW FUNCTIONS. UDFs can also be used in SELECT statements.
Fixed an issue where queries could be stuck in planning and accumulate until a coordinator restart is required.
Resolved an issue with queries against AWS Glue that were failing due to errors when loading an endpoints.json partitions file.
Fixed an issue where a reflection is given a score of 0 if an error occurs while calculating the score. Now the score will be empty instead of 0.
When no new data is read during REFRESH REFLECTION jobs, the snapshot IDs of the datasets and reflections that they depend on are shown in the Refresh Decision section of the query profile.
Improved logout functionality.
The Edit Rule dialog now auto-populates with information from the existing rule.
You can now open the Details Panel from the options menu on the Datasets page.
The result summary table now sorts cached query results in the summary table on the SQL Runner page in the order that the queries are executed.
You can now see the selected value for a reflection's partition transformation in the reflections editor.
Fixed a compilation issue that could occur when a window function is used with an ARRAY type column.
Fixed an issue that could occur when complex types are returned when splitting a function such as ARRAY_COMPACT.
Fixed an issue that could prevent a reflection score from being provided when running USE to set the query context.
Fixed an issue where a failed reflection could show an incorrect record count and size in the sys.reflections system table.
Fixed an issue that could cause ANALYZE TABLE to fail when table column names contained reserved keywords.
September 10, 2024
Fixed an intermittent issue that could cause project creation to fail with a ProjectConfigServiceException. Project creation is no longer prevented due to an interruption when an asynchronous source is being created or updated, causing the source to not update properly.
September 3, 2024
Fixed an issue that could result in a leak from an unclosed connection in Microsoft SQL Server, Oracle, or Dremio cluster data sources.
Fixed an issue that could cause VACUUM CATALOG to fail with a ContainerNotFoundException. Also fixed a bug that could cause VACUUM CATALOG to fail with an IllegalArgumentException if a view is created in an Arctic catalog.
August 22, 2024
Dremio now supports writes using merge-on-read in the Apache Iceberg table properties, which creates positional delete files and optimizes DML operations.
A reflection score shows the value that a reflection provides to your workloads based on the jobs that have been executed in the last 7 days.
For reflections on Iceberg tables, a new type of refresh policy is available. You can now automatically refresh reflections for underlying tables that are in Iceberg format when new snapshots are created after an update.
When reflection refresh jobs fail, Dremio now retries the refresh according to a uniform policy.
You can authenticate to a Snowflake source using key pair authentication.
User impersonation is now supported for Microsoft SQL Server sources.
OPTIMIZE TABLE now supports Iceberg tables with equality deletes.
Mapping table columns to the corresponding Parquet columns has been improved for Iceberg tables that are created from Parquet files and have columns without IDs.
Fixed an issue with long calls to AWS Glue sources that could result in a deadlock, preventing the Glue database from appearing as a source in the Dremio console and privileges granted to roles and users from applying properly to that source.
Fixed an issue that could prevent reflections with a row-access or column-masking policy from accelerating queries after an upgrade.
Automatically generated reflection recommendations now appear only if they meet a minimum threshold of value to your workloads.
In the reflections editor, the Refresh Now button no longer appears for failed reflections.
Clicking on a dataset on the Datasets page or clicking the Open Results link on the Job Overview page creates a new tab that is not automatically saved as a script.
Fixed an issue that could prevent reflections from being created for queries that contain an OVER clause with a specified RANGE.
Reduced memory usage when SELECT statements are run from the information schema by adjusting the page size parameter for pagination.
Fixed an issue that could cause the CURRENT_TIME function to return incorrect data when a user's timezone is defined.
Improved the query performance for VACUUM TABLE when using EXPIRE SNAPSHOTS.
Fixed an issue that could prevent partition columns from being applied in INSERT and CREATE TABLE AS statements.
August 12, 2024
Fixed a performance issue that affected queries containing a window function and a large number of batches.
MIN_REPLICAS and MAX_REPLICAS are no longer considered reserved keywords for SQL queries.
July 31, 2024
You can now use role-based access control (RBAC) privileges to restrict users and groups from accessing folders and their contents. With this change, admin users must explicitly grant visibility of folders and their contents to users and roles on the Arctic catalog as described in Arctic Privileges. To revert to the previous “open by default” behavior in which all objects are visible to all users in the PUBLIC role, see Inheritance.
For a given query with views, the reflection recommender now provides an aggregation reflection recommendation if possible instead of only default raw reflection recommendations.
AWS Glue lake formation permission cache can now be invalidated by users on demand by using ALTER SOURCE or the Source API. The lake formation tag policy support is also enabled by default.
Results caching improves query performance for non-UI queries with a result set that is less than 20 MB by reusing results for subsequent jobs with the same deterministic query and without underlying dataset changes. To use this feature, you must configure the time-to-live (TTL) rule in your project store to clear the cache.
Improved query planning time for over-partitioned tables with complex partition filters.
A query with an inner join can now match with reflections that contain outer joins.
Added a new Dataset API endpoint, POST /dataset/{id}/reflection/recommendation/{type}, for retrieving reflection recommendations by reflection type for a dataset.
The Catalog API Privileges endpoint is deprecated. We expect to remove it by July 2025.
In place of the Privileges endpoint, use the Catalog API Grants endpoint to retrieve privileges and grantees on specific catalog objects.
You can click Generate in the reflections tab to get a suggestion for creating an aggregation reflection. Statistics are no longer automatically collected and suggestions are generated when you open the reflections editor.
sys.project.pipe_summary is a new system table that summarizes high-level statistics for autoingest pipes and is only accessible to admins.
The flow of queries is no longer coupled with query telemetry, which means that failure scenarios in the flow could affect query completion rates. Queries now succeed despite any failures with query telemetry processing or JTS availability, even in the case of incomplete profile information.
Fixed an issue with concurrent dataset modifications that could cause jobs to hang during the metadata retrieval or planning. An inline metadata refresh is now retried automatically after a failure due to a concurrent source modification.
Fixed a bug for complex queries that could result in an error message about the code being too large.
Reflections have been fixed in the following ways:
-
The default selected columns for raw reflections no longer fail to include all columns of a dataset.
DX-89497 -
Queries no longer fail if an underlying default raw reflection becomes invalid for substitution against the view. The workaround is to disable or refresh the reflection.
DX-85139
If an autoingest pipe job has been canceled by a query engine, the pipe job now retries to ingest the canceled batch.
Fixed the following NullPointerExceptions (NPEs) that could occur:
-
When failed jobs details are fetched.
92934 -
When accessing large Delta Lake tables in metastore sources.
DX-67629 -
Where the schema for a Delta Lake table was not captured correctly, leading to a failure to query the table.
DX-92477 -
When running a DML statement on an accelerated table.
DX-91682
Queries no longer fail due to a ConcurrentModificationException when runtime filters are present.
Added a CONFIGURE BILLING privilege so that non-admin users can view and modify billing account data.
To prevent unexpected out-of-memory errors, the Parquet vectorized reader allocates only the necessary amount of memory for scanning deeply nested structures.
Fixed a performance issue for Iceberg tables that could occur when Dremio reads position delete files. Previously, a position delete file could be accessed multiple times by different scan threads. Now all delete rows are read once and joined with the data files.
Fixed a bug that could cause concurrent autopromotion of the same folder path to fail.
In the Dremio console, ideographic spaces now display as regular spaces in the results.
Fixed a bug in the SQL Runner where 
might not be visible in the Scripts panel for a script with a long name.
Fixed a bug where the commit history may not load for tables or views that reside in hyphenated folders.
The user avatar at the bottom of the left navigation bar now shows the user's first and last initials instead of the first two letters of their username.
When you are editing the preview engine in the Edit Engine dialog, the currently selected instance family is no longer shown in the notification at the top of the dialog.
Scripts have been fixed in the following ways:
-
Switching between scripts while a job is running no longer causes the job to appear in other tabs.
DX-92260 -
Opening a script and applying a transformation on a saved job should now work as expected.
DX-92754 -
Running a subset of a script now highlights the appropriate queries when switching between results tabs.
DX-92143
The reflection data in the job summary of a raw profile will now render successfully even when the accelerationDetails field is skipped.
CAST TIME AS VARCHAR now returns the result in 'HH:mm:ss.SSS' format.
You can now clear the context for the query session by running a USE command without any parameters.
The CONVERT_TIMEZONE function now works properly for Druid data sources.
LEAD and LAG functions with the window set to a value that is greater than 1 no longer produce incorrect results.
July 9, 2024
View schema learning now occurs only for queries that are issued from the Dremio console or reflection refresh jobs.
Queries no longer hang on coordinator startup when the materialization cache takes a long time to start up.
A raw profile is now available as soon as a job is in a running state.
Fixed a bug where duplicate rows could be returned when retrieving usage objects.
ORDER BY expressions in a subquery should be removed automatically as long as the query does not have LIMIT or OFFSET parameters, although the returned sort order cannot be guaranteed. In this example, ORDER BY deptno should be removed:
SELECT *
FROM emp
JOIN (SELECT * FROM dept ORDER BY deptno) USING (deptno)
Some databases like Postgres and Oracle support ORDER BY expressions, so you may see different results depending on the target of your query.
July 2, 2024
Reflection recommendations automatically generate for the top 10 most effective default raw reflections based on query patterns from the last 7 days. You can view these recommendations on the Reflections page in the Dremio console.
Added a retry mechanism when reflections are expanded into the materialization cache, which adds fault tolerance to coordinator upgrades and restarts.
User impersonation is now supported for Oracle sources.
The Privileges dialog is improved for managing sources, views, tables, and folders.
You can now bulk delete scripts.
You can specify a column as a MAP data type in CREATE TABLE.
You can use VACUUM CATALOG for Arctic sources on Azure.
Deleting a project in Standard edition no longer results in autoingestion being unavailable.
The usernames in Arrow Flight JDBC/ODBC and Legacy JDBC/ODBC jobs are now shown in the same consistent case regardless of the username case in the connection URL.
Fixed an issue that could introduce duplicate rows in the results for RIGHT and FULL joins with non-equality conditions and join conditions that use calculations.
Updated error messaging for creating or deleting a folder on non-branch references.
Updated the following library to address potential security issues:
- org.postgresql:postgresql to version 42.4.5 [CVE-2024-1597] DX-91055
When you query the information schema, you can now see only the tables and views that you have access to instead of all datasets.
Added a rule that pushes an aggregate below a join if the grouping key is also a join key.
All existing engines without an instance family have been backfilled to either m5d or ddv5 depending on the cloud vendor.
Correlated subqueries that include a filter that doesn't match any rows no longer result in an error message.
Reflection recommendations now occur when plan regeneration is required and the name of the dataset is not fully qualified and contains a period (for example, "arctic1"."@username@dremio.com".v1).
When a dataset is created in a source, the dataset inherits its owner from the source. Inheritance no longer fails if the source owner is inactive; instead, the dataset owner is now set to the system user.
The author ID no longer appears as the author's name in the commit history after a branch is merged using a SQL command.
Dataset version sorting no longer results in incorrect "not found" error messages when listing datasets in the Dremio console.
Reflections with row and column access control now produce the correct results when algebraically matched.
The current owner of a script is now correctly displayed in the Dremio console.
Certain font ligatures are no longer displayed in the results table on the SQL Runner page.
Disabling Download Query Profiles for admin and users now correctly restricts users from downloading profiles.
The raw query profile has been improved to include Execution Resources Planned and Execution Resources Allocated planning phases to help with debugging execution-related issues.
Users who do not have the required privileges to view all user and role names when using the Dremio console to manage privileges can add privileges by entering users' and roles' exact names in the Add User/Role field.
You can now use the Secret Resource URL when adding an Oracle source, which could not be used previously in an Oracle source due to a "missing password" error.
In the Advanced view of the reflections editor, you can select the SQL functions to use as measures in the Measure column for aggregation reflections.
The listing of catalog items no longer times out due to a very large number of catalog objects. To address the issue, optional pageToken and maxChildren parameters have been added to the API endpoints for getting catalog entities with children by ID or by path.
Indexing the same JSON into CONVERT_FROM multiple times no longer leads to incorrect results.
June 5, 2024
The Dremio JDBC driver now supports parameters in prepared statements.
You can use autoingest pipes to set up and deploy event-driven data ingestion pipelines directly in Dremio. This feature is in preview for Dremio Cloud and supports Amazon S3 as a source.
The retention period of jobs history stored in Dremio has been reduced from 30 days to 7 days, which improves job search response times. Use the jobs history system table to get the jobs history for jobs that are older than 7 days.
DML and CTAS are supported for the query_label workload management rule.
There are two new methods to start refreshing a reflection.
When an incremental refresh materialization is deprecated, you no longer see a DROP TABLE job in the job history but the reflection data is synchronously cleaned up as a part of reflection management.
For Azure projects, you can now create a table or view when the name of the table or view has a dot such as "arctic1"."@username@dremio.com".v1.
Users (including admin users) can now use the Scripts API to manage scripts from API clients for migration, management during owner offboarding, and other purposes.
All write operations for Arctic views are written in the new Iceberg Community View format (V1). Existing views are still supported in the old format (V0), although any update to an existing view rewrites the view in the new format. Read operations are supported for both V0 and V1. To see which view format is being used, open the Details panel or metadata card for the view. For Dialect, the V0 views show DREMIO and V1 views show DremioSQL.
ON CONFLICT and DRY RUN clauses are now available for MERGE BRANCH.
New SQL commands have been added for autoingest pipes: CREATE PIPE, ALTER PIPE, DESCRIBE PIPE, and DROP PIPE.
When a reflection that depends on certain file formats (Iceberg, Parquet, ORC, or Avro) is due for a refresh and has no new data, a full refresh is no longer performed where data is read in the REFRESH REFLECTION job. Instead, only a new refresh is planned and a materialization is created, eliminating redundancy and minimizing cost for the reflection.
Default raw reflection matching can now be used during REFRESH REFLECTION jobs.
Reflections are no longer deleted when a reflection refresh fails due to a network error or the source being down.
Duplicate default raw reflection recommendations are no longer created when querying a view that contains joins.
When multiple jobs are submitted to the reflection recommender, the reflection recommender no longer errors out if some of the jobs are ineligible for recommendation. Instead, reflections are recommended for eligible jobs.
TBLPROPERTIES (table properties) for Iceberg tables are now saved in Apache Hive.
Reading a Delta Lake table no longer fails with an error about an invalid Parquet file.
The AWS Lake Formation tag authorizer now considers database-level tags.
Dremio now honors workload management rules that contain the query_label function.
When using an IAM role and attempting to add an AWS Glue source, you no longer see an error message about loader constraint violation due to AWS Glue authentication.
Reflections no longer incorrectly match into queries containing ROLLUP.
On the Organization page, hovering over Learn more for Arctic and selecting the Get Started with Arctic link opens the updated Getting Started with Dremio page.
During the signup process, the catalog is no longer missing in the CloudFormation Template (CFT) parameters if the CFT failed the first time and you click Rerun CloudFormation template.
If you delete a branch or a tag that you are currently on, you are now rerouted to the Data page for the default reference instead of seeing an error message.
Tooltips on the Catalog page are now displayed correctly on Firefox.
Dataset names are no longer truncated incorrectly.
An error message no longer appears when loading results of multiple jobs that executed on different engines.
Error messages that appear when a user tries to view the wiki of the folder for which they don't have privileges now describe the problem more clearly.
Creating a new script while on a script that displays an error message no longer causes the error message to persist.
You can now use decimals in ARRAY_REMOVE and ARRAY_CONTAINS functions.
NPE has been fixed when ARRAY_CONTAINS is used in a WHERE clause.
New line characters (\n) are supported in regex matching.
Incorrect splitting no longer occurs when the value contains UNICODE characters like 'á'.