Skip to main content

Changelog

This changelog provides a detailed record of the previous 12 months of updates and enhancements we have made to improve your Dremio Cloud experience.

May 22, 2024

Issues Fixed
General Updates

The system_iceberg_tables plugin now uses the project ID in the project store in order to isolate data for each project.

DX-88657

May 14, 2024

What's New
General Updates

Integrating with AWS Lake Formation is now supported, which provides access controls for datasets in the AWS Glue Data Catalog and defines security policies from a centralized location that may be shared across multiple tools.

DX-66898,DX-86225,DX-87881,DX-87856

The query profile now contains the origin of the error such as COORDINATOR or EXECUTOR in addition to the error type that is prefixed to the error message. In case of "OUT_OF_MEMORY ERROR:" error types, the type of memory causing the error such as HEAP or DIRECT_MEMORY and the additional information about the current memory usage can now be seen in the verbose section of the error in the profile.

DX-87623

You can now use MongoDB as a data source in Dremio Cloud in Azure. For details, see MongoDB.

DX-62548,DX-87191

Call failures to the JTS when sending an intermediate executor profile are now ignored.

DX-86625

Reliability for Dremio coordinators has improved.

DX-86675,DX-71182

Privileges are now available for folders in Arctic catalogs, and CREATE FOLDER, CREATE TABLE, and CREATE VIEW privileges have been added for Arctic catalogs. Privileges are also now inherited for objects in Arctic catalogs.

DX-66080,DX-87080

Support has been added for reading Apache Iceberg tables with equality deletes.

DX-84522

The CSV reader now uses direct memory instead of heap memory.

DX-87586

Queries now succeed even if telemetry storage fails. While a query is running, the executors and the coordinator send telemetry about the query execution to the JTS, which is written to a persistent store when the query completes or fails. This incomplete telemetry is indicated in the Job Details page for transparency. is_profile_incomplete has been added in the system.project.jobs table to indicate the profile status and incomplete data.

DX-86907

In Enterprise edition, you can now select a secrets management option in place of existing secrets/password fields inside of the source creation and source edit dialogs.

DX-88797,DX-88824

APP

Out-of-the-box observability metrics are now available for user activity and jobs such as most active users, longest running jobs, most queried datasets, and more.

DX-86592,DX-83785

If the job profile results are incomplete, you are notified and the options to download the profile and see the raw profile are unavailable.

DX-86913

A Record fetch size has been added as a parameter in the settings for Snowflake sources. The default fetch size is 2000.

DX-87901

Arctic source pages are accessible for commits, tags, and branches. After you open an Arctic source, you can see Settings icon for settings in the top right of the Datasets page actions.

DX-88053,DX-88010,DX-88007

When creating an organization, you have additional role options and can select multiple roles.

DX-87945

API

A new sys.reflection_lineage table function lists all reflections that will be refreshed if a refresh is triggered for a particular reflection.

DX-69462,DX-85833

Reflection refreshes can now be configured based on a schedule. You can pick a time of day (UTC) and days of the week to refresh reflections for sources and tables.

DX-68532

Rate limits have been added to the Jobs service API on both the UI and Public API.

DX-84524,DX-88776

SQL

The authorTimestamp in SHOW LOGS has changed from VARCHAR to the dateTime data type.

DX-85299

Issues Fixed
General Updates

To get a count of the number of rows in a table, Dremio now requests an estimated document count rather than aggregating the document. As a result, Dremio can retrieve the count more quickly.

DX-88043

Catalog maintenance tasks are introduced for controlling and preventing duplicate dataset versions from being created by API calls. Two scheduled tasks now run daily:

  • Trimming the lists to a maximum of 50 records that are subject to a 30-day time-to-live (TTL) or the TTL value that is configured for jobs.
  • Removing temporary dataset versions generated by jobs that are older than 30 days or according to the configured TTL.
    DX-87659, DX-87549

Memory tracking issues that would cause queries to be cancelled due to exceeded the memory limits are fixed (with Memory Arbiter enabled and high memory utilization on the node).

DX-89359

Refresh Tokens Lifetime for Tableau is extended to 90 days to handle offline use cases like extracts.

DX-89195

Promoted datasets with inconsistent partition depth no longer occasionally throw an ArrayIndexOutOfBoundsException when filtering against deeper partitions.

DX-88365

A client pool has been added for more performant concurrent Hive metastore operations. Pool size can be controlled by store.hive3.client_pool_size, if set to 0 pooling is disabled.

DX-89821

Implementation has been added for the AWS Glue Data Catalog to pull and use Lake Formation tag policies. By default, this feature is turned off.

DX-87856

Commons-compress has been updated to version 1.26.1 [CVE-2024-25710] to address potential security issues.

DX-88906

An issue with slot assignment for preview engines is now fixed by adding RDBMS sources and metadata refresh for the same source type. Preview engines no longer have assigned slots that cannot be released, and new queries can get free slots and no longer have timeout errors.

DX-88883

Dremio now uses multiple writers in parallel for non-partitioned table optimization. The small files generated during the writing are combined by another round of writing with a single writer.

DX-88174

Reflections have been fixed in the following ways:

  • Reflections containing temporal functions such as NOW and CURRENT_TIME are no longer incrementally refreshed, potentially producing incorrect results. REFRESH REFLECTION jobs for reflections containing these dynamic functions are now full refreshes.
    DX-89451
  • Reflection refresh jobs no longer show zero planning time when the refresh is incremental.
    DX-87548
  • A snapshot-based incremental reflection refresh for unlimited-split datasets on Hive no longer results in excessive heap usage due to metadata access during the reflection refresh.
    DX-88194

Query profiles no longer show planning phases twice.

DX-89262

ROLE and USER audit events are now available in the sys.project.history.events table in the default project of an organization.

DX-90378

An issue no longer occurs if Lake Formation tag policies are present, but there are no Lake Formation tags defined on a certain table.

DX-89519

To prevent conflicts between SLF4J 1.x and 2.x, the Dremio JDBC driver no longer exposes the SLF4J API and uses the java.util.logging (JUL) framework to log messages. The application can be configured for the parent logger for the driver by using java.sql.Driver#getParentLogger() or directly using java.util.logging.Logger#getLogger("com.dremio.jdbc").

DX-56164

For AWS standard edition, users no longer see an unsupported error when clicking on a dataset to query the dataset.

DX-89390

Dremio blocks view creation if the view has a cyclic dependency on itself.

DX-86115

Hash join support structures are reallocated when they are insufficient for incoming batch.

DX-87903

If the wiki editor is empty when generating a summary, it is now automatically plugged into the wiki editor.

DX-86089

An intermittent failure when retrieving a wiki has been fixed.

DX-88638

API

For tables and views in a catalog, table and view owners need to have the USAGE privilege on the catalog to retrieve and create reflections. Previously, table or view ownership was sufficient to retrieve and create reflections using the reflections API.

DX-66080,DX-66095

accelerationNeverRefresh and accelerationNeverExpire are properly populated in /api/v3 for sources.

DX-87934

APP

The performance of loading the data visualization for jobs on the Monitoring page has improved.

DX-89276

The UI no longer breaks when clicking to set the refresh schedule for certain source types.

DX-90025

Privileges have been updated in the following ways:

  • If you have only the ALTER privilege on a dataset and no privileges on the catalog, you can open the folder and see the dataset, but you cannot edit the dataset or run the query.
    DX-87891
  • If you have only the SELECT privilege on a dataset and no privileges on the catalog, you can open the folder, see the dataset, and run a query on the dataset but you cannot edit the dataset.
    DX-87891
  • Users who do not belong to the ADMIN role cannot view the User filter or the list of users on the Jobs page.
    DX-87660
  • The view owner is now properly listed in the Dremio console after the owner is updated.
    DX-88705

The Table Cleanup tab in the catalog settings sidebar is no longer hidden when a user who belongs to the ADMIN role is viewing the Catalog Settings page with a non-admin role.

DX-88126

The SQL Runner has improved in the following ways:

  • When you run a query in the SQL runner, the page no longer briefly displays the previous query's results.
    DX-83509
  • The results of previously run queries now load much more quickly. After you open a saved script in the SQL Runner, the results are automatically displayed in a summarized format if at least one job in the script has successfully completed. To load the results of a specific query, select the query tab above the results table.
    DX-90110,DX-90627
  • Running a query and attempting to save it as a view no longer causes the results to disappear.
    DX-86266
  • Switching between the tabs in the SQL editor now correctly displays the job type.
    DX-89787
  • Script names are no longer prevented from being saved after users rename a tab and edit the SQL content.
    DX-86751
  • When expanding the large data field in the SQL Runner by using the ellipsis (...), the results are now responsive when the data includes DateTime objects.
    DX-86541
  • There is no longer a data correctness issue where joins with non-equality conditions and join conditions using calculations would sometime introduce duplicate rows (while respecting desired filtering properties) into the result set.
    DX-90720
  • Background threads no longer run when a query in the SQL Runner is cancelled or fails.
    DX-85812

Adding a folder to the primary catalog now uses the references of the primary catalog rather than the selected source. When adding a folder to the primary catalog using the + icon, the folder is now correctly added to the primary catalog and not to the selected Arctic source.

DX-88708,DX-89191

The Details panel is no longer blank when you open the panel from the hover menu inside of an Arctic source.

DX-87923

The dataset metadata card no longer incorrectly opens versioned views from the Job Details page.

DX-88778

Messaging has been improved for:

  • notifying if the top level space or source doesn't exist during view creation
    DX-85784
  • successfully updating script privileges when using GRANT TO ROLE or GRANT TO USER
    DX-88527
  • failed queries with row-access and column-masking policies using reflections
    DX-88480
  • failed metadata cleaning due to the expiration of snapshots
    DX-69750
  • switching between tabs in the SQL Runner
    DX-87980
  • originating from Hive3 plugin engine for the cases where other source types are served, such as Hive 2.x or AWS Glue.
    DX-87596
  • notifying you if the catalog for the source has been deleted or cannot be found when you attempt to use a source
    DX-65235
  • attempting to create a table or view that has the same path as an existing folder
    DX-86880

Merge action in the Dremio console no longer shows the user ID instead of the user email in the commit.

DX-88377

SQL

The SKIP_FILE option for the COPY INTO SQL command no longer fails to handle Parquet file corruption issues if the issues are in the first page of a row group.

DX-87884

Trailing semicolons that terminate an OPTIMIZE TABLE statement are no longer marked as an error in the SQL Runner.

DX-85903

Scalar UDFs no longer returns incorrect results in some cases.

DX-87600

A reliability issue in ARRAY_AGG has been fixed.

DX-87000,DX-87384

DML no longer breaks after a table is recreated and merged.

DX-87984

You can now perform DML queries when no context is selected.

DX-88172

April 16, 2024

What's New
General Updates

Azure images for Dremio Cloud executors have been upgraded from CentOS 8.5 Gen 2 Linux to the Ubuntu 22.04 Linux distribution.

DX-86588

April 2, 2024

What's New
General Updates

Enabled the memory arbiter by default in order to monitor the usage of four key operators: HASH_AGGREGATE, HASH_JOIN, EXTERNAL_SORT, and TOP_N_SORT. This usage is monitored across all queries running on an executor to improve how the executor utilizes its direct memory and to reduce OutOfMemoryException errors.

  • If the memory arbiter detects that the memory usage is too high, then the memory usage will be reduced in these two ways:
    • Starting with the biggest consumers, some of these operators will need to reduce their memory usage mainly by spilling to disk.
    • Memory allocations will be blocked.

Enabled HASH_JOIN to spill to disk by default when the memory allocated for a query is fully utilized.

DX-48798

Added support for column mapping within Delta Lake tables, effectively supporting minReaderVersion 2.

DX-62046

Improved partition pushdowns for Iceberg tables when queries use datetime filters that include a function on a column.

DX-65078

Improved coordinator startup times by allowing Dremio to serve queries while the materialization cache is loading in the background. Some queries might not be accelerated during this time.

DX-83342

Made it possible find out the oldest reflection data used by accelerated queries by looking for "Last Refresh from Table" in the job summary of a raw profile or by querying the system table SYS.PROJECT.REFLECTIONS and looking for last_refresh_from_table in the output.

DX-84702

Disabled C3 caching during loads of Parquet source files via the COPY INTO operation, thereby reducing cache contention with other query workloads.

DX-85365

Reduced the heap memory used by the SORT operator.

DX-53594

Improved reliability and memory efficiency for Dremio coordinators.

DX-86245

Reading Iceberg tables with positional deletes no longer causes an IndexOutOfBoundsException.

DX-87252

Improved Dremio's capabilities for concurrent DML operations on Iceberg tables and improved error messaging for concurrent load failures.

DX-85437

Updated the Arrow package version to 14.0.2 to include Dremio Arrow fixes, and to include new features and fixes from Apache Arrow.

DX-85890

Added support for Version 15 of the Arrow Flight JDBC driver. You can download the driver from here.

Added support for the copy_errors() table function on Parquet tables.

DX-87332

Added support for limiting access to specified databases on Glue sources.

DX-87812 DX-88223 DX-88420 DX-87811

API

Improved the Projects API in these two ways:

  • The Project object now includes the lastStateError object for projects for which Dremio encounters an "invalid credentials" error.
  • The Projects API can be used to update project credentials.
    DX-65288

Added to Reflection Summary objects of the Reflection API and the SYS.PROJECT.REFLECTIONS table the error message that explains the most recent failure of a refresh of a reflection. No message appears if no refresh has yet been attempted, no failure has occurred, or a successful refresh has followed a failed one.

DX-85499

Reflection recommendations are now associated with the corresponding job IDs.

DX-86672

Added the parameters isMetadataExpired and lastMetadataRefreshAt to the Table and View objects of the Catalog API. Now, when either of these two methods is called and a table or view has stale or no metadata, there is no automatic refresh of the metadata:

GET /v0/projects/{project-id}/catalog/{id}*

GET /v0/projects/{project-id}/catalog/by-path/{path}

Instead, users can look at the values for the two new parameters and decide whether to invoke a refresh by calling this method:

POST /v0/projects/{project-id}/catalog/{id}/refresh
DX-86745
APP

Changed the tabs in the SQL runner to display the most recent results of a query, if the results are available from the job history, without the user having to run the query again.

DX-85843

Added support for editing the credentials of projects that use AWS or Azure. This can be done in the Project Storage section of a project's settings.

DX-86740 DX-86760

Privilege changes are processed more quickly in the Dremio console.

DX-87547

SQL

Added a check to determine whether users running the COPY INTO command have SELECT privileges on either the source storage location specified in the FROM clause or on each individual source file mentioned in the FILES clause.

DX-85977

Performance is improved and memory consumption is reduced for some INFORMATION_SCHEMA queries that filter on TABLE_NAME or TABLE_SCHEMA.

DX-87580

Issues Fixed
General Updates

Fixed planning errors that resulted from accessing views on which a reflection that included the CONVERT_FROM() function was defined.

DX-59542

Fixed an issue that caused an additional empty subnet field to appear in the second step of the process for creating the first project of an organization. This field appeared if a cloud tag was added during the first step.

DX-63663

Fixed an issue that allowed reflections to be created when their definitions included UDFs that contained context-sensitive functions.

DX-86078

Fixed an issue that prevented full exception stack traces from being provided for errors generated by queries that included an ambiguous column name.

DX-86233

Queries are no longer cancelled due to exceeding memory limits while spilling during a SORT operation.

DX-49883

Fixed an issue with case sensitivity that would lead to delayed processing of inherited privileges.

DX-88683

Fixed an issue with orphaned materialization datasets in the catalog due to incremental reflection refreshes that were not writing any data.

DX-54184

When an incremental reflection refresh is skipped due to no data changes, there is no longer an issue where the last refresh finished, causing the last refresh from the table and last refresh duration not to update in the reflection system tables and reflection summary.

DX-84572

Fixed an issue that caused filters on scans of source tables on MongoDB to use incorrect regex.

DX-86902

Ensured that group policy grants are respected in AWS Lake formation when Dremio is used with Okta.

DX-86923

Fixed an issue that occurred if "All tables" was selected during AWS Lake formation and the granting of a new permission that was meant to apply to all tables within the selected database.

DX-86925

APP

Fixed an issue that caused the Save as View option in the SQL editor not to work after the option had already been used once in a single session.

DX-85904

Fixed an issue that prevented the Details icon from appearing for items in lists of sources or lists of spaces in the Dremio console.

DX-86337

Fixed an issue preventing users from accessing the Edit wiki button in the Details Panel.

DX-87005

Navigating between datasets using the lineage graph in the Dremio console no longer results in a message about unsaved changes.

DX-87876

Fixed an issue that caused the maximum number of scripts that could be saved in the Dremio console to be 100, not 1,000.

DX-86540

Fixed an issue that caused the creation of a new branch to update the context of the SQL Runner automatically.

DX-87039

Fixed an issue that prevented attempts to join datasets from the SQL Runner not to correctly default to the Custom Join tab when no recommended join existed.

DX-87044

Fixed an issue that caused the Delete Organization dialog to appear after an organization had been renamed and the new name saved.

DX-87046

Fixed an issue that enabled a Save button when an added subnet field had not been filled in.

DX-87389

Fixed an issue that caused the AWS Secret Access Key field to appear blank even when a key was specified. Now, the masked key is visible.

DX-87403

SQL
Fixed the handling of SQL functions, such as LOWER, UPPER, and REVERSE, in queries on system tables.
DX-52626

Made more concise the error messages for schema mismatch errors that occur for UNION ALL operators.

DX-85966

Dremio no longer caches CURRENT_DATE_UTC and CURRENT_DATE during query planning, which was causing incorrect results. As a result, queries that use CURRENT_DATE_UTC and CURRENT_DATE have some performance latency in favor of accurate results.

DX-86078

Fixed an issue that caused the SQL function APPROX_COUNT_DISTINCT to return null instead of 0 in some cases.

DX-86597

February 26, 2024

Issues Fixed
General Updates

A valid location has been provided for DoGet requests, resolving a compatibility issue with the Arrow Flight JDBC 15 driver and ADBC driver.

DX-85225

February 12, 2024

Issues Fixed
General Updates

Privilege changes are processed more quickly in the Dremio console.

DX-87547

February 1, 2024

Issues Fixed
General Updates

An exception error no longer occurs when you run a query or create a view with an ambiguous column name.

DX-87220

January 31, 2024

What's New
General Updates

Incremental refreshes can be performed on reflections that are defined on views that use joins.

DX-84768
DX-85818

The Reflection Recommender gives recommendations to users with complex queries and deep semantic layers for better performance and predictable matching. The queries for which default raw reflections can be recommended must run against one or more views that match certain criteria.

DX-84616

General Updates

You can now use MongoDB as a data source in Dremio Cloud in AWS. For details, see MongoDB.

DX-62548

General Updates

For Azure Blob Storage and Data Lake Gen 2 sources, checksum-based verification is enabled to ensure data integrity during network transfers.

DX-66932

APP

You can click Close Others to close all tabs besides the active tab in the SQL editor.

DX-86642

SQL

The ARRAY_FREQUENCY function is now supported.

DX-67298

Issues Fixed
General Updates

Creating a raw reflection on a dataset on which no reflections are already defined no longer creates an aggregation reflection.

DX-86098

To alter the reflections on a view or table, the user or role must have the privilege ALTER_REFLECTION on it and also have the USAGE and COMMIT privileges on the Arctic catalog.

DX-86425

General Updates

Query planning times are shorter during the metadata validation phase due to view schema learning.

DX-86504

There is no longer an exception during the planning of queries on views that use the INTERVAL data type.

DX-86315

General Updates

Queries against Iceberg tables with positional deletes no longer fail with an error like “the current delta should always be larger than the amount of values to skip."

DX-85123

General Updates

Unneeded columns are now trimmed from JDBC pushdowns.

DX-85187

General Updates

The performance of health checks of AWS Glue data sources has been improved with checks of the state of the metastore and attempts to retrieve databases with a specified maximum result limit for 1.

DX-85641

General Updates

The successful generation of labels and wikis no longer requires an engine to be running.

DX-83266

APP

Selectively-run queries will be highlighted as errors, if they fail.

DX-85901

APP

The dialog that explains a query has failed should no longer appear when you are switching between SQL tabs.

DX-86514

APP

When adding a new Arctic catalog source fails, the error message now provides detailed information about the specific error.

DX-84510

SQL

Previously, if you used a statement in your query to set a schemapath to an Arctic source and folder, then the table or view validation would fail. Now, you can set the context to an Arctic source that includes any number of folders.

DX-85412

January 16, 2024

What's New
General Updates

Reflections on views that join two or more anchor tables (Apache Iceberg tables and certain types of datasets in filesystem sources, Glue sources, and Hive sources) can now be refreshed incrementally.

DX-84768

Dremio now uses Micrometer-based metrics. Existing Codahale-based metrics are preserved and include the tag {metric_type="legacy"}.

DX-85268

Executor metrics tags now include engineId and subEngineId.

DX-86497

API

You can use the Recommendations API to submit job IDs of jobs that ran SQL queries, and receive recommendations for aggregation reflections that can accelerate those queries. See Recommendations for more information.

DX-68447

SQL

These terms were added to the list of reserved keywords: JSON_ARRAY, JSON_ARRAYAGG, JSON_EXISTS, JSON_OBJECT, JSON_OBJECTAGG, JSON_QUERY, and JSON_VALUE.

DX-86250

Known Issues
Known Issues

The following words were incorrectly made reserved keywords: ABSENT, CONDITIONAL, ENCODING, ERROR, FORMAT, PASSING, RETURNING, SCALAR, UNCONDITIONAL, UTF8, UTF16, UTF32.

Issues Fixed
APP

Fix a bug causing archived sonar projects not to appear for a user on the Sonar Projects page immediately after that user received the Admin privilege.

DX-62382

A NullPointerException could be returned when a row count estimate could not be obtained.

DX-69646

The tutorials that are accessed from the left navigation bar are available only to the creators of organizations, not to all users of organizations.

DX-83091

The settings for configuring a new catalog no longer appear until the cloud or type of cloud is chosen.

DX-84521

The Add Column, Group By, and Join buttons could be disabled if the SELECT command that defined a view was run and that command ended in a semicolon.

DX-84535

If you saved a new view in the SQL Runner and then re-opened the SQL Runner, the view that you had just created would still be present.

DX-84536

For some types of data sources, the generation of a wiki page would fail.

DX-84627

The Save button for reflections defined on views in spaces would be enabled for public users who have only SELECT, EDIT, and VIEW REFLECTION privileges. Such users still were correctly prevented from modifying reflections, as clicking Save did nothing.

DX-84684

Reflection management orphaned reflection materialization tables that were in the KV store. These tables would never get cleaned up and cause the KV store to become larger than necessary.

DX-84804

Querying Apache Druid tables containing large amounts of data could cause previews in the SQL Runner to time out.

DX-84983

All columns were being sent in JDBC predicate pushdowns.

DX-85187

Queries with correlated subqueries could return incorrect results.

DX-85193

An exception occurred when Dremio tried to get an estimate of the row count for PostgreSQL tables.

DX-85303

Opening the SQL Runner from the Details page of a table caused the SQL Runner to open with the SQL editor hidden in the new tab and in all open tabs.

DX-85372

Scrolling through phases and operators in a visual profile was sometimes jumpy.

DX-85534

Users without permission to edit a view in an Arctic source were able to access the view's SQL definition if a direct URL to the Detail page for the view was provided by a user who did have edit permission.

DX-85558

The wrong branch could become active after you refreshed the SQL Runner page and then clicked on the breadcrumbs at the top of that page.

DX-85575

If you clicked a view or a table, ran the generated SELECT * statement in the SQL Runner, and then clicked the Edit button in the dataset details on the right, the SQL Runner would not be refreshed with the DDL for creating the view or table. The SQL statement and successful/failed query in the SQL runner will remain in the editor page when navigating to a dataset.

DX-85828

API

In API requests to create a new project, the catalogName body parameter is now required.

DX-85158

December 14, 2023

What's New
General Updates

You can now add an Azure private endpoint in the Azure portal when you connect your Azure account to Dremio Cloud or add a project to an organization. The outbound private endpoints are used to connect Dremio executors to the Dremio Cloud control plane over the Azure network.

DX-52453

The Dremio-to-Dremio connector is now supported in Azure.

PM-2107

Automated table cleanup to delete expired snapshots and orphaned metadata files is now supported for Iceberg tables in Arctic catalogs.

DX-63305

The algorithm that triggers a refresh of dependent reflections has been improved to prevent duplicate refreshes. The refresh operation now remains in a pending state until all direct and indirect dependences finish refreshing.

DX-82601

For reflections that are defined on Parquet datasets in S3 sources, Dremio can now automatically choose incremental refresh or full refresh.

DX-67226

Planning time for reflections has been substantially improved. The acceleration profile now contains a detailed breakdown of reflection normalization and substitution times.

DX-68120

The external token provider audit log now includes audit events for creating and updating BI applications.

DX-83652

API

The Clouds API now includes the privateEndpoints parameter for specifying an Azure private endpoint.

DX-69135

APP

You can now use tabs in the SQL Runner to work on multiple tasks simultaneously. All of your work in each tab is autosaved.

DX-68951

The Visual Profile now displays notable observations and potential problems for operators and phases. Users can use filters to control which operators are displayed.

DX-84695
DX-83760
DX-84178
DX-84006

The Visual Profile now shows the following runtime metrics: waitTimeSkew, wallClockTimeSkew, batchesProcessedSkew, sleepingDuration and cpuWaitTime.

DX-84180

When users try to edit a deleted script, they will now see a confirmation dialog with the following options to prevent lost work: Discard, Copy SQL, and Save as script.

DX-84785

SQL

This update adds support for the following SQL functions: ARRAY_AGG, ARRAY_APPEND, ARRAY_DISTINCT, ARRAYS_OVERLAP, ARRAY_PREPEND, ARRAY_SLICE.

DX-65490
DX-68069
DX-68082
DX-68612

Known Issues
Known Issues

If you disable the Query dataset on click setting, the Datasets page does not include The Go to Table icon for tables and views. To query a dataset, click The SQL Runner icon in the left navigation panel to open the SQL Runner or click The More icon for the dataset and select Query.

Issues Fixed
General Updates

Users can now set privileges on folders with a . character in their names and the tables these folders contain.

DX-84934

Iceberg metadata table functions no longer truncate the number of results returned to the maximum batch size set for exec.batch.records.max.

DX-84328

Row-level runtime filtering is disabled for reflection refresh jobs so that views no longer return incorrect results due to an incorrect match to a single Starflake reflection.

DX-68144

When connecting to an Apache Druid source, the username and password are now optional.

DX-84444

When modifying the credentials for an existing Arctic catalog, the external ID for the IAM role now persists rather than refreshing with the page.

DX-68779

View schema learning has been improved to handle complex types and no longer requires query re-planning.

DX-84117

Fixed a NullPointerException (NPE) that occurred during split assignment of Delta Lake scans.

DX-60247

When creating recommendation reflections, more than one recommendation may be created in response to a single job ID. Also, the initial SQL query can now contain outer joins that are part of a view definition, in addition to inner joins, and set operators. See Reflection Recommendations for more information.

DX-68447

Updated Calcite to version 1.19.

DX-68808

APP

When a user logs out, all UI context is now cleared.

DX-65026

Logging out while on the Settings page for an Arctic catalog no longer results in an error.

DX-84330

All scripts are now visible when users scroll to the end of the scripts list in the SQL Runner. Also, the displayed number of scripts is now accurate up to 1000.

DX-84659
DX-84641

In the SQL Runner functions panel, the filter categories are now listed in alphabetical order.

DX-84840

In the SQL Runner, the copy button is now disabled while queries are running.

DX-84711

Using the tab character in object names no longer causes inconsistent column spacing.

DX-85027
DX-84608

On the Job Overview page for a canceled query, clicking the View Profile tab no longer results in an error.

DX-85170

The Job Overview page no longer reports incorrect state information for reflections.

DX-67339

A new script is no longer created when you open the SQL Runner by clicking a dataset name and then click the back button to return to the previous screen.

DX-85277

When users are on the Job Details page, the browser tab name now correctly displays Job Details - Dremio.

DX-84560

For queries with a large number of results, truncation messages now display the correct number of rows of results.

DX-85077

When deleting a script, users now receive only a single confirmation dialog.

DX-84612

Table results now clear correctly when users save a run or previewed query as a script.

DX-84782

When editing a query, users can now see the previewed results of a transformation on the previously selected dataset.

DX-84726

The APPROX_COUNT_DISTINCT function now properly calculates the approximate count distinct rather than the exact count distinct.

DX-84197

SQL

Fixed an issue where queries that contain correlated subqueries in the join condition could return duplicate rows.

DX-83748

Queries that involve array columns that contain string values no longer fail.

DX-82518

Fixed a performance issue that affected queries that contain many GET calls for large arrays.

DX-61308

A balanced UnionAll subtree now prevents stack issues when inserting a large number of values.

DX-84042

In some cases, the HASH_JOIN operator could request more memory at the beginning of its work than anticipated. When this happens, instead of allowing the query to fail, Dremio now satisfies the operator's request and takes note of the elevated memory requirement.

DX-83720

Users now receive a more informative error message for ALTER TABLE queries that attempt to set a masking policy that refers to a non-existent function.

DX-83696

November 27, 2023

What's New
APP

You can connect your Azure account to Dremio Cloud when getting started or adding a project or cloud to your organization for the following supported regions: East US, Central US, and West Europe. Learn more about the Azure prerequisites and how you can get started.

DX-65305

SQL

The COPY INTO command now supports Parquet files.

DX-68624

Issues Fixed
APP

You no longer need the MONITOR privilege to run Arctic optimization jobs.

DX-84929

November 16, 2023

What's New
APP

You can see a view definition or an Arctic table definition if you have the SELECT privilege, although editing a view definition requires further privileges.

DX-68952

APP

You can now see syntax errors in your SQL query as you enter the query into the SQL editor. Each error is automatically detected with a red wavy underline and contains information about the type of error. For more information, see Syntax Error Highlighting.

DX-82679

APP

The details panel can be collapsed so it no longer overlaps the SQL Runner page or Datasets page, making it easier to access and to use for switching between details for different objects.

DX-82691

SQL

Dremio now supports the SQL commands SHOW CREATE VIEW to see a view definition and SHOW CREATE TABLE to see a table definition. For more information, see SHOW CREATE VIEW and SHOW CREATE TABLE.

DX-68952

SQL

The following SQL functions are now supported: ATAN2, BITWISE_AND, BITWISE_NOT, BITWISE_OR, BITWISE_XOR, DATETYPE, HASH64, PARSE_URL, PMOD, STRING_BINARY, and TIMESTAMPTYPE.

DX-72648

Issues Fixed
APP

Folders are no longer deleted from the main branch when using the delete folder option.

DX-83363

APP

When using hash joins, queries no longer fail with unexpected restart of an executor.

DX-63029

APP

The default job results cleanup path no longer results in disk space issues and unexpected restarts on some cluster nodes.

DX-41291

APP

In the new source dialog for Arctic sources, the following configuration options have been moved from the Storage tab to the Advanced Options tab: Disable check for expired metadata while querying and Enable source to be used with other sources even though Disable Cross Source is configured.

DX-71969, DX-83976

APP

When hovering over a very long label for a dataset in the details panel, the label name is no longer cut off in the tooltip.

DX-83264

APP

When generated labels are a subset of existing labels for a dataset, the Append button is disabled inside the dialog.

DX-83210

APP

Previously, if a user dropped a branch in which reflections were created, the reflections defined by the datasets on that branch would not be deleted in the next reflection refresh cycle. Those reflections would become orphaned and never get cleaned up. This issue is now fixed.

DX-83819

APP

For Hive and Glue sources, filters are now successfully pushed down to the Iceberg Manifest Scan.

DX-83260

APP

The parsing of CSV files has become more strict. Quoted values are now expected to be terminated properly with the quote symbol before reaching the end of the file; otherwise, an UnmatchedQuoteAtEOFException will be thrown.

DX-82894

APP

Extra columns in a CSV file (compared to target table schema) no longer cause issues during a COPY INTO ON_ERROR ('continue') job.

DX-82891

APP

Query profile now shows the correct resolved table/key count when a SQL context is set in a query or view.

DX-68291

APP

Users can now browse tables in catalogs whose names include an underscore.

DX-66487

APP

Billing and usage views now more accurately reflect Azure-specific engine characteristics.

DX-53677

APP

Role endpoints that are PUBLIC now return limited information. These endpoints are called by UI in the context of searching a role or getting the role information.

DX-83060

APP

The visual profile is no longer prevented from working in some cases due to strict security measures.

DX-84343

APP

Operations to add a row-access policy no longer fail because the UDF couldn't be resolved.

DX-83610

APP

If a query used in a reflection contains a UDF, reflection refreshes no longer fail with a plan serialization error.

DX-66628

APP

In order to increase coordinator stability, the plan cache size has been decreased from 10k queries to 1k queries and the time duration from 10 days to 8 hours.

DX-83190

SQL

For datasets created by Dremio, the CREATE TABLE, REFRESH REFLECTION, OPTIMIZE TABLE, and INSERT INTO SQL commands will now have dictionary encoding enabled. If the page data lends itself to dictionary encoding, the corresponding page data will be dictionary encoded.

DX-61238

SQL

Error handling is improved when users create a view with a full query starting with CREATE VIEW.

DX-83094

SQL

The reflection recommender now provides user queries that include COUNT(DISTINCT) and/or APPROX_COUNT_DISTINCT with accurate reflection recommendations.

DX-69989

SQL

Handling of inferred partition columns is improved. Specifically, FOR PARTITIONS (...) now works properly for inferred partition columns.

DX-60565

October 31, 2023

What's New
APP

Removed an errant dependency check that was preventing some engines from starting or scaling replicas.

DX-84081

APP

Fixed an issue with AWS regional STS endpoint support for Glue sources that assume an AWS role. To enable AWS region STS endpoint support, set value of the property fs.s3a.assumed.role.sts.endpoint to the STS endpoint hostname for the region that you are using. For example, the value might be sts.us-east-1.amazonaws.com.

DX-84114
APP

Metadata on AWS Glue sources was not being refreshed according to the schedule defined on the source. In some cases, new data was only seen after ALTER TABLE <table> REFRESH METADATA was run.

DX-82900

Known Issues
Known Issues

Due to metadata caching, it may take up to five minutes to reflect revoked privileges on objects in a Sonar project, including on Arctic catalogs.

Known Issues

Users with the organization-level MANAGE GRANTS privilege who have not been assigned the ADMIN role are not able to assign privileges to users or roles unless they have been explicitly assigned the CREATE USER or CREATE ROLE privilege.

DX-62435

October 23, 2023

What's New
APP

VACUUM CATALOG, which removes expired snapshots and orphaned metadata files for Iceberg tables, is now supported in Dremio Cloud. For Arctic catalogs, you can configure automatic table cleanup and set the cutoff policy in catalog settings. Dremio uses the cutoff policy to determine which snapshots and associated files to expire and delete. For Arctic catalog sources, you can manually run VACUUM CATALOG on demand. For more information, see Enabling Table Cleanup and Setting the Cutoff Policy and VACUUM CATALOG.

APP

You can now use Dremio Cloud's Generative AI capabilities to create wikis and labels for datasets. For more information, see Generative AI.

DX-71209, DX-71208

APP

In the advanced editor for reflections, Dremio now can recommend partition columns.

DX-83532

APP

Reflection and query plan caches are now cleared when they are disabled to ensure that queries do not use a deprecated reflection.

DX-83117

APP

Arctic catalog settings, details pages, and API responses now recommend new URL patterns with Nessie API version v2 preselected.

DX-82734

APP

Improved validation of the S3 root path when adding an Arctic source.

DX-82697

APP

Plans for queries containing CONVERT_FROM(JSON) can now be cached.

DX-82619

APP

Text-to-SQL events are now available in system history tables.

DX-79949

APP

Updated the operation used to refresh Delta Lake table metadata to improve performance.

DX-71691

APP

The sys.organization.usage system table now returns usage data for 365 days instead of 90 days.

DX-71094

APP

For new projects created after October 23rd, users must be assigned the USAGE privilege on the project before they can access or execute queries against any resource within the project's scope. For projects that existed before October 23rd, users who are members of the PUBLIC role automatically have the USAGE privilege on the project. For more information, see Project Visibility and Access.

DX-70886

APP

Tooltips have been added to disabled copy buttons in the case that you are viewing a page over HTTP instead of HTTPS.

DX-68256

APP

Partition recommendations for reflections based on a single partitioned table are now available.

DX-68214

APP

Incremental refresh query plans have been optimized to avoid Iceberg metadata scans when the snapshotID has not changed since the last refresh.

DX-66604

APP

If a source owner is removed from Dremio, another user with permission to the source can now promote datasets and change the source configuration in place of the owner that was removed.

DX-58562

SQL

Dremio Cloud now supports the ARRAY_TO_STRING SQL function, which returns a string of the values provided in the input array. For more information, see ARRAY_TO_STRING.

Dremio Cloud now supports the SET_UNION SQL function, which returns a single array that includes all of the elements from the input arrays, without duplicates. For more information, see SET_UNION.

Issues Fixed
API

In some cases, the billing API was returning incorrect data for account balances.

DX-67674

APP

Dremio was unable to read and query AWS Glue table partitions if partition column names or partition values contained spaces or other special characters.

DX-83517

APP

When adding a project to an existing cloud, the CFT flow was ignoring the selected AWS region and directing to US_WEST_2 instead.

DX-83398

APP

Fixed an issue that was causing an exception during filter pushdown into a Parquet scan.

DX-82982

APP

Nessie sources with names that included special characters were not loading properly in the Dremio console.

DX-82979

APP

In some cases, incremental reflection refresh by partition was resulting in truncated data when the base dataset and the reflection used truncate Iceberg transform.

DX-82978

APP

The metadata card was not showing up if you hovered over a dataset with a forward-slash in its name.

DX-82868

APP

When viewing details for a versioned dataset, the History tab was not displaying any information.

DX-82864

APP

The Columns section in the dataset details panel was not updating if you selected a different dataset without first closing the details panel.

DX-82788

APP

In some cases, unnecessary warnings about metadata changes were being displayed when editing Arctic source properties.

DX-82684

APP

Fixed some minor scrolling and table display issues on the Project Settings > Engines page.

DX-71185

APP

If a dataset name was the same as one of the tabs in the dataset details view (data, details, reflections, history), clicking to edit the dataset or clicking the Go to Table button would take you directly to the tab with the same name as the dataset.

DX-70978

APP

For some browsers, an interruption in connectivity can cause a failure in updating the status of long-running queries.

DX-69909

APP

Default raw reflections could not be substituted into a query that used UNION with mixed types, which was causing longer than normal planning times.

DX-69653

APP

In some cases, running ALTER TABLE <table_path> FORGET METADATA against a view could result in the view being deleted instead of the command failing with an error.

DX-68202

APP

Fixed an issue that was preventing users from creating aggregation reflections without dimensions via SQL, even though such reflections could be created in the Dremio console.

DX-67967

APP

Fixed the following issues with acceleration information in job profiles when the plan cache was used: acceleration information was missing for a prepared query, plan cache usage was missing for a prepared query, acceleration information was missing when the query was not accelerated but reflections were considered, and canonicalized user query alternatives were missing. Additionally, matching hints were missing for reflections that were only considered.

DX-64636

APP

If a date pattern only contained the year and month, the parsed date was returned as the last day of the previous month instead of the first day of the specified month.

DX-24037

SQL

In some scenarios, when Query dataset on click was enabled, clicking on a dataset was opening the dataset in the SQL Runner with an empty query instead of a default SELECT statement on the dataset.

DX-83367

October 12, 2023

What's New
APP

Dremio Cloud now supports access to cross-account S3 and Glue data sources in VPCs that utilize private subnets. To enable this access, the following connection properties must to be added under Source Settings > Advanced Options:

For S3 sources:

  • fs.s3a.assumed.role.sts.endpoint = sts.<aws-region>.amazonaws.com
  • fs.s3a.endpoint = s3.<aws-region>.amazonaws.com

For Glue sources:

  • aws.region = <aws-region>

Prior to adding any sources, an S3 gateway VPC endpoint and an STS interface VPC endpoint must be created in the VPC.

October 11, 2023

Issues Fixed
APP

Users who were not assigned to the ADMIN role were unable to run queries against tables and views that did not have an owner. Owners were missing from tables and views created prior to the August 17 update of Dremio Cloud. For some tables and views, an error scenario could have caused the owner to be missing.

DX-83526

October 6, 2023

Issues Fixed
APP

Users could drop a table or view from an Arctic catalog if they had USAGE and COMMIT privileges on the catalog and SELECT privileges on the table or view. With this update, only users with USAGE and COMMIT privileges on the catalog and OWNERSHIP privileges on a table or view, or users in the ADMIN role, can drop a table or view.

DX-83294

APP

In the Dremio console, it appeared as though a user without OWNERSHIP privileges on an Arctic catalog could delete the catalog, even though they could not.

DX-83294

SQL

In some cases, queries that used CONVERT_FROM in a filter condition were failing.

DX-82945

September 21, 2023

What's New
APP

Dremio Arctic and all of its related features are no longer in preview mode.

DX-82613

APP

In the Dremio console, ownership in Sonar and Arctic is now listed separately from other privileges, at the top of the Privileges page, and the procedure for transferring ownership is streamlined. For more information, see Transferring Organization Ownership and Transferring Ownership.

DX-69442

APP

Added keyboard shortcuts in the SQL Runner for showing or hiding the Text-to-SQL panel and for triggering Text-to-SQL. For more information, see Keyboard Shortcuts.

DX-67771

APP

The Record Count column has been moved next to the Current Footprint column in the Project Settings > Reflections table.

DX-60641

APP

Added support for the il-central-1: Israel (Tel Aviv) region to the AWS Glue source.

DX-69481

APP

Dremio Cloud provides more helpful information in the error message if an invalid tag or branch name is supplied.

DX-69034

APP

You can now leave the Database Name field blank in a PostgresSQL source.

DX-67472

APP

Reduced the number of S3 lookups required for Arctic DML and DDL operations to improve performance in query planning.

DX-66247

APP

Reduced the amount of heap memory used by the query plan cache.

DX-69868

Issues Fixed
API

Updated the Source API to prevent sending secret values in clear text.

DX-44436

APP

Setting query_label() as an engine rule was resulting in an exception error.

DX-71949

APP

In some cases, default raw reflection matching was not working as expected for users not assigned to the ADMIN role.

DX-70893

APP

After searching for and selecting a username or role in catalog privileges, the search string was not automatically being cleared.

DX-68966

APP

The Tableau and Power BI buttons were visible in the SQL Runner for unsaved queries.

DX-69867

SQL

Saving a query as a view without having run the query was resulting in an error if no engine replicas were active.

DX-71680

SQL

Fixed an issue with COL_LIKE() when input and pattern contained the % character.

DX-69931

SQL

LIKE was not being highlighted as a reserved keyword in the SQL editor.

DX-69755

SQL

Fixed an issue that was causing the use of GRANT ALL on a project to fail with an "invalid project privilege" error.

DX-65571

SQL

OPTIMIZE and VACUUM queries on tables with reflections were being evaluated for reflection matching, causing an error.

DX-64997

SQL

Fixed an issue with filter pushdowns that was causing some preview queries to fail even though the same query was successful when using Run.

DX-56539

September 12, 2023

Issues Fixed
SQL

Fixed an issue that was causing the RST_STREAM closed stream error when processing large result sets via JDBC or ODBC.

DX-64679

September 8, 2023

Issues Fixed
SQL

Some queries were failing with a Failure getting source error.

DX-71093

SQL

Queries against views and tables in an Arctic catalog were not showing up on the Jobs page. This issue only affected organizations that use the default Arctic catalog instead of spaces.

DX-67652

September 6, 2023

What's New
APP

The following regions are now supported in the AWS Glue source:

  • ap-south-2: Asia Pacific (Hyderabad)
  • ap-southeast-3: Asia Pacific (Jakarta)
  • ap-southeast-4: Asia Pacific (Melbourne)
  • eu-south-2: EU (Spain)
  • eu-central-2: EU (Zurich)
  • me-central-1: Middle East (UAE)
    DX-69347
SQL

Added a new table function, SYS.RECOMMEND_REFLECTIONS, that recommends aggregation reflections to accelerate existing SQL queries. For more information, see Reflection Recommendations.

DX-61857

SQL

This update adds support for the following SQL functions:

Issues Fixed
APP

The Status of some failed queries was being reported as RUNNING instead of FAILED in the Jobs Overview page.

DX-69767

APP

Updated com.google.guava:guava to 32.1.1-jre to address CVE-2023-2976 in Dremio's internal Iceberg fork.

DX-68888

APP

Updated validation settings to ensure that only privileged users could view Acceleration Settings on the Project Settings > Reflections page.

DX-68877

SQL

Fixed an issue that was causing inconsistent query results when ARRAY_CONTAINS was used with nullability checks.

DX-69920

SQL

Plans for queries containing CONVERT_FROM could not be cached.

DX-66675

SQL

To address a CONCURRENT_MODIFICATION error seen in concurrent metadata refresh queries on Parquet tables, if the query is submitted by scheduler, failures are ignored. If the query is submitted by users, the failed query is retried until it succeeds.

DX-69749

August 28, 2023

What's New
APP

When viewing catalog or folder contents on the Datasets page, the "More" menu (...) for tables and views now contains a Delete option, allowing users with appropriate privileges to delete a table or view.

DX-68290

Issues Fixed
APP

For some failed queries, status in the Job profile was being reported as RUNNING instead of FAILED.

DX-69767

APP

When creating a new project, the Arctic catalog name was not being validated prior to launching the CloudFormation template.

DX-69621

APP

The Open Results link on the Jobs overview page was not working as expected for queries that were run from edit mode on the Dataset page.

DX-69601

APP

When saving a view, the items in the "Save View As" dialog were not sorted in the same way as on the Dataset page.

DX-68432

SQL

For organizations created prior to August 17, 2023, granting or revoking table or view privileges for the first time via SQL was successful, but an error was produced on the Organization Settings > Privileges page.

DX-69645

SQL

If a dataset name or the name of a parent folder contained a space or ampersand (&) character, clicking on the dataset would populate the SQL Runner with a truncated SELECT statement.

DX-66366

August 17, 2023

What's New
APP

As of this update, each Sonar project in new Dremio Cloud organizations will come deployed with an Arctic catalog, which will support data management capabilities (folders, tables, etc.) for the project. This primary Arctic catalog replaces your home space.

DX-63250

APP

You can now use Role-Based Access Control (RBAC) privileges to control which roles and users can read, write, and manage tables and views in Arctic catalogs.

DX-42564

APP

Dremio Sonar now supports the same SQL syntax as Spark when working with Arctic/Nessie sources.

DX-60108

APP

Dremio automatically optimizes incremental reflection file size to improve reflection performance.

DX-61779

APP

In this update, you must explicitly create folders (namespaces) before creating tables or views in them.

DX-67970

APP

The Usage page under Organization Settings now includes usage data for Arctic catalogs.

DX-66291

APP

Dremio will avoid a full data scan for simple aggregations on partition columns, reading the manifest metadata instead, which improves performance for queries on very large tables.

DX-66029

SQL

After DML operations against unpartitioned Iceberg tables, Dremio now compacts the data files written by the DML operation to improve future read performance.

DX-58884

Issues Fixed
APP

Updated the Snowflake connector to fix intermittent issues when adding Snowflake as a source.

DX-67959

APP

If you optimized a table in Dremio Arctic (Optimize Once) and then viewed the dataset settings for another table, the Optimize Once button remained disabled unless you refreshed the page.

DX-67808

APP

Logged in users were getting redirected to the login page instead of the create password page when clicking on an invite that had not yet been accepted.

DX-67398

APP

When running a job multiple times, the status and job link for the last attempt are now displayed as expected.

DX-66716

APP

Increased concurrency limits to avoid errors when concurrent inserts into the same table were being sent from different streams.

DX-61285

SQL

Fixed an issue that could cause a memory leak when querying an Iceberg table with positional deletes.

DX-68497

SQL

At times, the DAY() function was returning either integer or timestamp, depending on how the query was written.

DX-67793

SQL

Fixed an issue that was causing an error when running OPTIMIZE TABLE on a table with reflections.

DX-67861

SQL

In some cases, the IF EXISTS option for DROP BRANCH and DROP TAG was being ignored.

DX-67553

SQL

Top-level CASE statements intended to return a Boolean were not being rewritten correctly, resulting in an error for some SQL Server queries.

DX-67527

SQL

Some SQL Server queries with nested CASE statements were failing with invalid SQL comparison syntaxes.

DX-67183

SQL

Fixed an issue with the LEFT() SQL function on Oracle sources for queries with dates.

DX-67036

SQL

Some date subtraction queries were not getting pushed down for Oracle sources.

DX-65396

July 27, 2023

Issues Fixed
SQL
Fixed an issue that was resulting in incorrect results when a view containing a correlated sub-query was used more than once in the same query.
DX-67662
SQL
Some reflections on views based on MySQL sources were failing with timeout errors.
DX-61201

July 24, 2023

What's New
APP

This update provides performance improvements in the Jobs listing page, and any user with sufficient privileges can now view reflection jobs in the table.

DX-64196

APP

The details panel displaying Wiki content is now available inside a folder on the main branch in an Arctic source.

DX-67453

APP

This update adds support for a new connector that allows querying data from Apache Druid. For more information, see Apache Druid.

DX-46607

SQL

You can now drag and drop a table from your home space into the Text-to-SQL panel in the SQL Runner.

DX-67329

SQL

We have made some improvements in the efficiency of the queries suggested when using the Text-to-SQL feature.

DX-66337

Issues Fixed
API
The Engines REST API was producing an error instead of generating an empty list if the user did not have access to any engines.
DX-65193
APP
If you optimized a table in Dremio Arctic (Optimize Once) and then viewed the dataset settings for another table, the Optimize Once button remained disabled unless you refreshed the page.
DX-67808
APP
If the preview engine had been disabled, it can now be enabled again.
DX-67282
APP
Unknown organizations were showing up when signing in to Dremio Cloud from client applications.
DX-67273
APP
A new external ID was being generated every time a cloud entity was edited, which was breaking the IAM role trust relationship. In such cases, users had to temporarily switch to IAM access key credentials with the same permissions as the IAM role.
DX-61964
APP
When parsing CSV, empty lines at the top of the CSV are now ignored, allowing column names to be extracted from lines other than the first
DX-60797
SQL
Fixed an issue that was allowing non-privileged users to alter or view reflections via SQL.
DX-67672

July 7, 2023

What's New
General Updates

Dremio Cloud has new onboarding flow that walks you through creating your first project and gives you an overview of Dremio features. You can hide the tutorials if you like, and show them again from the Help menu.

June 29, 2023

What's New
APP

This update enables new privileges for Arctic catalog-level role-based access control: COMMIT, CREATE BRANCH, CREATE TAG, MANAGE GRANTS, OWNERSHIP, USAGE, and WRITE SETTINGS.

Breaking Change: With this change, non-privileged users (e.g., users who are not catalog or organization owners) will not be able to access Arctic catalogs and their underlying datasets. Catalog or organization owners will need to explicitly grant these privileges to users and roles on the Arctic catalog, as described in Privileges.

APP

The metadata card is now displayed for datasets in Arctic catalogs. For more information, see Viewing the Metadata of an Arctic Dataset.

APP

You can now build reflections on versioned Arctic tables and views. For more information, see Using Arctic Branches with Reflections.

APP

The "Data-as-code" source category has been renamed "Arctic Catalogs."

APP

You can now add the SampleDB source to any project using the Add Data Source dialog.

APP

The "Profile" tab on the Jobs detail page has been renamed "Visual Profile."

APP

You can now add wiki content and labels to Dremio Arctic catalogs that are available in your Sonar project (main branch only).

SQL

Autocomplete has been improved to provide better and faster context-aware suggestions for SQL keywords, catalog objects, and functions while you are constructing SQL statements.

SQL

When dragging a dataset into the SQL editor, the AT BRANCH | TAG | COMMIT <reference> will be appended to the query.

SQL

Dremio now supports LIST literals using the ARRAY keyword. For more information, see Data Types.

SQL

The schema for the sys.organization.usage table has been changed to support Arctic catalog usage. The project_id column has been split into entity_id and entity_type so you can quickly determine if usage belongs to a Sonar project or an Arctic catalog. For more information, see SYS.ORGANIZATION.USAGE.

Issues Fixed
APP

Added validation to ensure that you cannot create an Engine Routing rule with the same name as an existing rule.

APP

The Preview engine cannot be disabled or deleted because it is used to run metadata refresh queries for relational sources and other background activities.

APP

Fixed a security issue that could, in some cases, allow users to bypass email verification for new accounts.

APP

Improved the message that is displayed when an Engine Routing rule rejects a query or if the query does not match any rules.

June 14, 2023

What's New
APP

Dremio's Text-to-SQL is now available in Preview! Check out Analyzing Data with Text-to-SQL for an example.

DX-63998

June 7, 2023

What's New
API

You can now use the REST API to cancel a Dremio Arctic optimization job. For more information, see Canceling a Job.

DX-63324
APP

You can now cancel a Dremio Arctic optimization job from the Catalog Settings > Configuration page. For more information, see Canceling a Job.

DX-65852
APP

The Project Settings page now includes an option to manage preferences for the Dremio console. The SQL settings option is no longer available, and the Autocomplete and Copy or Download Results settings are now available under Preferences. For more information, see Dremio Preferences.

DX-65852
APP

You can now configure the default behavior in the Dremio console when clicking on a dataset to either open the SQL Runner with a default SELECT statement on the dataset or open the Datasets page. For more information, see Querying a Dataset on Click.

DX-65852
APP

The metadata card for datasets now includes links to launch a BI tool. For more information about the metadata card, see Viewing the Metadata of a Dataset.

APP

This update includes changes that provide improved performance for planning times for queries on Dremio Arctic views.

DX-66218
APP

The name and type of a dataset's source is now displayed when viewing the dataset in the lineage graph.

DX-65153
APP

Added more security around DML permission checks to ensure that users cannot access data to which they should not have access.

DX-64746
Issues Fixed
APP

After modifying and saving the details of a user and selecting another user to edit, the details of the previously modified user were being displayed very briefly in the user detail fields.

DX-66513
APP

Some reflection refreshes that were accelerated by default raw reflections were not being matched in user queries.

DX-66212
APP

Users who had not been assigned to the ADMIN role could see the Add Space button when viewing all spaces.

DX-65186
APP

In some cases, invalid partition stats were causing the row count estimate for a table to come back as 1.

DX-64986
APP

In the dataset details panel, wiki content was being truncated if it was too long. You can now scroll horizontally on wiki details.

DX-64091
APP

At times, during background metadata refresh, some folders on Amazon S3 sources that had been formatted as tables were losing their formatting. This was causing the PLAN ERROR: UnsupportedOperationException error for external dashboards that used views or reflections built on those tables.

DX-63415
APP

Empty, disabled reflections were being created if you edited an existing reflection.

DX-63356
APP

When using AWS PrivateLink and handling large amounts of data, system table queries could exceed the default timeout, resulting in a FlightRuntimeException error.

DX-63087
APP

Optimization jobs were initially being displayed at the bottom of the Jobs list, but they will now be included in the list according to start time.

DX-62569
APP

When promoting some datasets, quoted fields were not being treated as quoted when white space was included before it.

DX-60798
SQL

Queries on History tables were failing intermittently.

DX-65439
SQL

GRANT commands on catalog entities were failing with "Role/User \<ID> not found" if existing user or role grantees were no longer present in the system.

DX-65364
SQL

In some cases, when clicking the Functions button in the SQL Runner, the list of functions was empty.

DX-66706
SQL

When opening a versioned view in a new tab, the Dremio console was displaying an error instead of loading the dataset.

DX-65857
SQL

Right-clicking on a dataset and selecting "Open in New Tab" was opening a new browser tab, but the SQL Runner was empty.

DX-65754

June 1, 2023

What's New
APP

When you add the first Sonar project to your Dremio Cloud organization, a new sample source called "SampleDB" is now included in the project. To get started with Dremio and learn how to query data across multiple sources and more, check out Getting Started with Dremio Sonar.

May 24, 2023

Issues Fixed
SQL

Right-clicking on a dataset and selecting "Open in New Tab" was opening a new browser tab, but the SQL Runner was empty.

DX-65754
SQL

Queries on sys.project.history tables were failing intermittently.

DX-65439
SQL

When clicking a dataset to query it, the "Unsaved Changes" dialog was being displayed in the SQL Runner after changes had already been saved or when no changes had been made.

DX-64579
SQL

When using AWS PrivateLink and handling large amounts of data, system table queries could exceed the default timeout, resulting in a FlightRuntimeException error.

DX-63087