Changelog
This changelog provides a detailed record of the previous 12 months of updates and enhancements we have made to improve your Dremio Cloud experience.
February 26, 2024
A valid location has been provided for DoGet
requests, resolving a compatibility issue with the Arrow Flight JDBC 15 driver and ADBC driver.
February 12, 2024
Privilege changes are processed more quickly in the Dremio console.
February 1, 2024
An exception error no longer occurs when you run a query or create a view with an ambiguous column name.
January 31, 2024
Incremental refreshes can be performed on reflections that are defined on views that use joins.
The Reflection Recommender gives recommendations to users with complex queries and deep semantic layers for better performance and predictable matching. The queries for which default raw reflections can be recommended must run against one or more views that match certain criteria.
You can now use MongoDB as a data source in Dremio Cloud in AWS. For details, see MongoDB.
3XS and 4XS engine sizes are now supported for smaller workloads that run Sonar queries or optimize Arctic catalogs.
For Azure Blob Storage and Data Lake Gen 2 sources, checksum-based verification is enabled to ensure data integrity during network transfers.
You can now refresh reflections by using an API method, ALTER TABLE
, and ALTER VIEW
. You can also refresh reflections on views by using the Catalog API.
You can click Close Others to close all tabs besides the active tab in the SQL editor.
The ARRAY_FREQUENCY
function is now supported.
Creating a raw reflection on a dataset on which no reflections are already defined no longer creates an aggregation reflection.
To alter the reflections on a view or table, the user or role must have the privilege ALTER_REFLECTION
on it and also have the USAGE
and COMMIT
privileges on the Arctic catalog.
Query planning times are shorter during the metadata validation phase due to view schema learning.
There is no longer an exception during the planning of queries on views that use the INTERVAL
data type.
Queries against Iceberg tables with positional deletes no longer fail with an error like “the current delta should always be larger than the amount of values to skip."
Unneeded columns are now trimmed from JDBC pushdowns.
The performance of health checks of AWS Glue data sources has been improved with checks of the state of the metastore and attempts to retrieve databases with a specified maximum result limit for 1.
The successful generation of labels and wikis no longer requires an engine to be running.
Selectively-run queries will be highlighted as errors, if they fail.
The dialog that explains a query has failed should no longer appear when you are switching between SQL tabs.
When adding a new Arctic catalog source fails, the error message now provides detailed information about the specific error.
Previously, if you used a statement in your query to set a schemapath to an Arctic source and folder, then the table or view validation would fail. Now, you can set the context to an Arctic source that includes any number of folders.
January 16, 2024
Reflections on views that join two or more anchor tables (Apache Iceberg tables and certain types of datasets in filesystem sources, Glue sources, and Hive sources) can now be refreshed incrementally.
Dremio now uses Micrometer-based metrics. Existing Codahale-based metrics are preserved and include the tag {metric_type="legacy"}
.
Executor metrics tags now include engineId
and subEngineId
.
You can use the Recommendations API to submit job IDs of jobs that ran SQL queries, and receive recommendations for aggregation reflections that can accelerate those queries. See Recommendations for more information.
These terms were added to the list of reserved keywords: JSON_ARRAY
, JSON_ARRAYAGG
, JSON_EXISTS
, JSON_OBJECT
, JSON_OBJECTAGG
, JSON_QUERY
, and JSON_VALUE
.
The following words were incorrectly made reserved keywords: ABSENT
, CONDITIONAL
, ENCODING
, ERROR
, FORMAT
, PASSING
, RETURNING
, SCALAR
, UNCONDITIONAL
, UTF8
,
UTF16
, UTF32
.
Fix a bug causing archived sonar projects not to appear for a user on the Sonar Projects page immediately after that user received the Admin privilege.
A NullPointerException could be returned when a row count estimate could not be obtained.
The tutorials that are accessed from the left navigation bar are available only to the creators of organizations, not to all users of organizations.
The settings for configuring a new catalog no longer appear until the cloud or type of cloud is chosen.
The Add Column, Group By, and Join buttons could be disabled if the SELECT command that defined a view was run and that command ended in a semicolon.
If you saved a new view in the SQL Runner and then re-opened the SQL Runner, the view that you had just created would still be present.
For some types of data sources, the generation of a wiki page would fail.
The Save button for reflections defined on views in spaces would be enabled for public users who have only SELECT, EDIT, and VIEW REFLECTION privileges. Such users still were correctly prevented from modifying reflections, as clicking Save did nothing.
Reflection management orphaned reflection materialization tables that were in the KV store. These tables would never get cleaned up and cause the KV store to become larger than necessary.
Querying Apache Druid tables containing large amounts of data could cause previews in the SQL Runner to time out.
All columns were being sent in JDBC predicate pushdowns.
Queries with correlated subqueries could return incorrect results.
An exception occurred when Dremio tried to get an estimate of the row count for PostgreSQL tables.
Opening the SQL Runner from the Details page of a table caused the SQL Runner to open with the SQL editor hidden in the new tab and in all open tabs.
Scrolling through phases and operators in a visual profile was sometimes jumpy.
Users without permission to edit a view in an Arctic source were able to access the view's SQL definition if a direct URL to the Detail page for the view was provided by a user who did have edit permission.
The wrong branch could become active after you refreshed the SQL Runner page and then clicked on the breadcrumbs at the top of that page.
If you clicked a view or a table, ran the generated SELECT *
statement in the SQL Runner, and then clicked the Edit button in the dataset details on the right, the SQL Runner would not be refreshed with the DDL for creating the view or table. The SQL statement and successful/failed query in the SQL runner will remain in the editor page when navigating to a dataset.
In API requests to create a new project, the catalogName
body parameter is now required.
December 14, 2023
You can now add an Azure private endpoint in the Azure portal when you connect your Azure account to Dremio Cloud or add a project to an organization. The outbound private endpoints are used to connect Dremio executors to the Dremio Cloud control plane over the Azure network.
The Dremio-to-Dremio connector is now supported in Azure.
Automated table cleanup to delete expired snapshots and orphaned metadata files is now supported for Iceberg tables in Arctic catalogs.
The algorithm that triggers a refresh of dependent reflections has been improved to prevent duplicate refreshes. The refresh operation now remains in a pending state until all direct and indirect dependences finish refreshing.
For reflections that are defined on Parquet datasets in S3 sources, Dremio can now automatically choose incremental refresh or full refresh.
Planning time for reflections has been substantially improved. The acceleration profile now contains a detailed breakdown of reflection normalization and substitution times.
The external token provider audit log now includes audit events for creating and updating BI applications.
The Clouds API now includes the privateEndpoints
parameter for specifying an Azure private endpoint.
You can now use tabs in the SQL Runner to work on multiple tasks simultaneously. All of your work in each tab is autosaved.
The Visual Profile now displays notable observations and potential problems for operators and phases. Users can use filters to control which operators are displayed.
The Visual Profile now shows the following runtime metrics: waitTimeSkew
, wallClockTimeSkew
, batchesProcessedSkew
, sleepingDuration
and cpuWaitTime
.
When users try to edit a deleted script, they will now see a confirmation dialog with the following options to prevent lost work: Discard, Copy SQL, and Save as script.
This update adds support for the following SQL functions: ARRAY_AGG
, ARRAY_APPEND
, ARRAY_DISTINCT
, ARRAYS_OVERLAP
, ARRAY_PREPEND
, ARRAY_SLICE
.
If you disable the Query dataset on click setting, the Datasets page does not include for tables and views. To query a dataset, click in the left navigation panel to open the SQL Runner or click for the dataset and select Query.
Users can now set privileges on folders with a .
character in their names and the tables these folders contain.
Iceberg metadata table functions no longer truncate the number of results returned to the maximum batch size set for exec.batch.records.max
.
Row-level runtime filtering is disabled for reflection refresh jobs so that views no longer return incorrect results due to an incorrect match to a single Starflake reflection.
When connecting to an Apache Druid source, the username and password are now optional.
When modifying the credentials for an existing Arctic catalog, the external ID for the IAM role now persists rather than refreshing with the page.
View schema learning has been improved to handle complex types and no longer requires query re-planning.
Fixed a NullPointerException (NPE) that occurred during split assignment of Delta Lake scans.
When creating recommendation reflections, more than one recommendation may be created in response to a single job ID. Also, the initial SQL query can now contain outer joins that are part of a view definition, in addition to inner joins, and set operators. See Reflection Recommendations for more information.
Updated Calcite to version 1.19.
When a user logs out, all UI context is now cleared.
Logging out while on the Settings page for an Arctic catalog no longer results in an error.
All scripts are now visible when users scroll to the end of the scripts list in the SQL Runner. Also, the displayed number of scripts is now accurate up to 1000.
In the SQL Runner functions panel, the filter categories are now listed in alphabetical order.
In the SQL Runner, the copy button is now disabled while queries are running.
Using the tab character in object names no longer causes inconsistent column spacing.
On the Job Overview page for a canceled query, clicking the View Profile tab no longer results in an error.
The Job Overview page no longer reports incorrect state information for reflections.
A new script is no longer created when you open the SQL Runner by clicking a dataset name and then click the back button to return to the previous screen.
When users are on the Job Details page, the browser tab name now correctly displays Job Details - Dremio
.
For queries with a large number of results, truncation messages now display the correct number of rows of results.
When deleting a script, users now receive only a single confirmation dialog.
Table results now clear correctly when users save a run or previewed query as a script.
When editing a query, users can now see the previewed results of a transformation on the previously selected dataset.
The APPROX_COUNT_DISTINCT function now properly calculates the approximate count distinct rather than the exact count distinct.
Fixed an issue where queries that contain correlated subqueries in the join condition could return duplicate rows.
Queries that involve array columns that contain string values no longer fail.
Fixed a performance issue that affected queries that contain many GET calls for large arrays.
A balanced UnionAll subtree now prevents stack issues when inserting a large number of values.
In some cases, the HASH_JOIN operator could request more memory at the beginning of its work than anticipated. When this happens, instead of allowing the query to fail, Dremio now satisfies the operator's request and takes note of the elevated memory requirement.
Users now receive a more informative error message for ALTER TABLE queries that attempt to set a masking policy that refers to a non-existent function.
November 27, 2023
You can connect your Azure account to Dremio Cloud when getting started or adding a project or cloud to your organization for the following supported regions: East US, Central US, and West Europe. Learn more about the Azure prerequisites and how you can get started.
The COPY INTO command now supports Parquet files.
You no longer need the MONITOR privilege to run Arctic optimization jobs.
November 16, 2023
You can see a view definition or an Arctic table definition if you have the SELECT
privilege, although editing a view definition requires further privileges.
You can now see syntax errors in your SQL query as you enter the query into the SQL editor. Each error is automatically detected with a red wavy underline and contains information about the type of error. For more information, see Syntax Error Highlighting.
The details panel can be collapsed so it no longer overlaps the SQL Runner page or Datasets page, making it easier to access and to use for switching between details for different objects.
Dremio now supports the SQL commands SHOW CREATE VIEW
to see a view definition and SHOW CREATE TABLE
to see a table definition. For more information, see SHOW CREATE VIEW and SHOW CREATE TABLE.
The following SQL functions are now supported: ATAN2
, BITWISE_AND
, BITWISE_NOT
, BITWISE_OR
, BITWISE_XOR
, DATETYPE
, HASH64
, PARSE_URL
, PMOD
, STRING_BINARY
, and TIMESTAMPTYPE
.
Folders are no longer deleted from the main branch when using the delete folder option.
When using hash joins, queries no longer fail with unexpected restart of an executor.
The default job results cleanup path no longer results in disk space issues and unexpected restarts on some cluster nodes.
In the new source dialog for Arctic sources, the following configuration options have been moved from the Storage tab to the Advanced Options tab: Disable check for expired metadata while querying and Enable source to be used with other sources even though Disable Cross Source is configured.
When hovering over a very long label for a dataset in the details panel, the label name is no longer cut off in the tooltip.
When generated labels are a subset of existing labels for a dataset, the Append button is disabled inside the dialog.
Previously, if a user dropped a branch in which reflections were created, the reflections defined by the datasets on that branch would not be deleted in the next reflection refresh cycle. Those reflections would become orphaned and never get cleaned up. This issue is now fixed.
For Hive and Glue sources, filters are now successfully pushed down to the Iceberg Manifest Scan.
The parsing of CSV files has become more strict. Quoted values are now expected to be terminated properly with the quote symbol before reaching the end of the file; otherwise, an UnmatchedQuoteAtEOFException
will be thrown.
Extra columns in a CSV file (compared to target table schema) no longer cause issues during a COPY INTO ON_ERROR
('continue') job.
Query profile now shows the correct resolved table/key count when a SQL context is set in a query or view.
Users can now browse tables in catalogs whose names include an underscore.
Billing and usage views now more accurately reflect Azure-specific engine characteristics.
Role endpoints that are PUBLIC
now return limited information. These endpoints are called by UI in the context of searching a role or getting the role information.
The visual profile is no longer prevented from working in some cases due to strict security measures.
Operations to add a row-access policy no longer fail because the UDF couldn't be resolved.
If a query used in a reflection contains a UDF, reflection refreshes no longer fail with a plan serialization error.
In order to increase coordinator stability, the plan cache size has been decreased from 10k queries to 1k queries and the time duration from 10 days to 8 hours.
For datasets created by Dremio, the CREATE TABLE
, REFRESH REFLECTION
, OPTIMIZE TABLE
, and INSERT INTO
SQL commands will now have dictionary encoding enabled. If the page data lends itself to dictionary encoding, the corresponding page data will be dictionary encoded.
Error handling is improved when users create a view with a full query starting with CREATE VIEW
.
The reflection recommender now provides user queries that include COUNT(DISTINCT)
and/or APPROX_COUNT_DISTINCT
with accurate reflection recommendations.
Handling of inferred partition columns is improved. Specifically, FOR PARTITIONS (...)
now works properly for inferred partition columns.
October 31, 2023
Removed an errant dependency check that was preventing some engines from starting or scaling replicas.
Fixed an issue with AWS regional STS endpoint support for Glue sources that assume an AWS role. To enable AWS region STS endpoint support, set value of the property fs.s3a.assumed.role.sts.endpoint
to the STS endpoint hostname for the region that you are using. For example, the value might be sts.us-east-1.amazonaws.com
.
Metadata on AWS Glue sources was not being refreshed according to the schedule defined on the source. In some cases, new data was only seen after ALTER TABLE <table> REFRESH METADATA
was run.
Due to metadata caching, it may take up to five minutes to reflect revoked privileges on objects in a Sonar project, including on Arctic catalogs.
Users with the organization-level MANAGE GRANTS
privilege who have not been assigned the ADMIN
role are not able to assign privileges to users or roles unless they have been explicitly assigned the CREATE USER
or CREATE ROLE
privilege.
October 23, 2023
VACUUM CATALOG
, which removes expired snapshots and orphaned metadata files for Iceberg tables, is now supported in Dremio Cloud. For Arctic catalogs, you can configure automatic table cleanup and set the cutoff policy in catalog settings. Dremio uses the cutoff policy to determine which snapshots and associated files to expire and delete. For Arctic catalog sources, you can manually run VACUUM CATALOG
on demand. For more information, see Enabling Table Cleanup and Setting the Cutoff Policy and VACUUM CATALOG.
You can now use Dremio Cloud's Generative AI capabilities to create wikis and labels for datasets. For more information, see Generative AI.
In the advanced editor for reflections, Dremio now can recommend partition columns.
Reflection and query plan caches are now cleared when they are disabled to ensure that queries do not use a deprecated reflection.
Arctic catalog settings, details pages, and API responses now recommend new URL patterns with Nessie API version v2 preselected.
Improved validation of the S3 root path when adding an Arctic source.
Plans for queries containing CONVERT_FROM(JSON)
can now be cached.
Text-to-SQL events are now available in system history tables.
Updated the operation used to refresh Delta Lake table metadata to improve performance.
The sys.organization.usage
system table now returns usage data for 365 days instead of 90 days.
For new projects created after October 23rd, users must be assigned the USAGE
privilege on the project before they can access or execute queries against any resource within the project's scope. For projects that existed before October 23rd, users who are members of the PUBLIC role automatically have the USAGE
privilege on the project. For more information, see Project Visibility and Access.
Tooltips have been added to disabled copy buttons in the case that you are viewing a page over HTTP instead of HTTPS.
Partition recommendations for reflections based on a single partitioned table are now available.
Incremental refresh query plans have been optimized to avoid Iceberg metadata scans when the snapshotID has not changed since the last refresh.
If a source owner is removed from Dremio, another user with permission to the source can now promote datasets and change the source configuration in place of the owner that was removed.
Dremio Cloud now supports the ARRAY_TO_STRING
SQL function, which returns a string of the values provided in the input array. For more information, see ARRAY_TO_STRING.
Dremio Cloud now supports the SET_UNION
SQL function, which returns a single array that includes all of the elements from the input arrays, without duplicates. For more information, see SET_UNION.
In some cases, the billing API was returning incorrect data for account balances.
Dremio was unable to read and query AWS Glue table partitions if partition column names or partition values contained spaces or other special characters.
When adding a project to an existing cloud, the CFT flow was ignoring the selected AWS region and directing to US_WEST_2 instead.
Fixed an issue that was causing an exception during filter pushdown into a Parquet scan.
Nessie sources with names that included special characters were not loading properly in the Dremio console.
In some cases, incremental reflection refresh by partition was resulting in truncated data when the base dataset and the reflection used truncate Iceberg transform.
The metadata card was not showing up if you hovered over a dataset with a forward-slash in its name.
When viewing details for a versioned dataset, the History tab was not displaying any information.
The Columns section in the dataset details panel was not updating if you selected a different dataset without first closing the details panel.
In some cases, unnecessary warnings about metadata changes were being displayed when editing Arctic source properties.
Fixed some minor scrolling and table display issues on the Project Settings > Engines page.
If a dataset name was the same as one of the tabs in the dataset details view (data, details, reflections, history), clicking to edit the dataset or clicking the Go to Table button would take you directly to the tab with the same name as the dataset.
For some browsers, an interruption in connectivity can cause a failure in updating the status of long-running queries.
Default raw reflections could not be substituted into a query that used UNION
with mixed types, which was causing longer than normal planning times.
In some cases, running ALTER TABLE <table_path> FORGET METADATA
against a view could result in the view being deleted instead of the command failing with an error.
Fixed an issue that was preventing users from creating aggregation reflections without dimensions via SQL, even though such reflections could be created in the Dremio console.
Fixed the following issues with acceleration information in job profiles when the plan cache was used: acceleration information was missing for a prepared query, plan cache usage was missing for a prepared query, acceleration information was missing when the query was not accelerated but reflections were considered, and canonicalized user query alternatives were missing. Additionally, matching hints were missing for reflections that were only considered.
If a date pattern only contained the year and month, the parsed date was returned as the last day of the previous month instead of the first day of the specified month.
In some scenarios, when Query dataset on click was enabled, clicking on a dataset was opening the dataset in the SQL Runner with an empty query instead of a default SELECT
statement on the dataset.
October 12, 2023
Dremio Cloud now supports access to cross-account S3 and Glue data sources in VPCs that utilize private subnets. To enable this access, the following connection properties must to be added under Source Settings > Advanced Options:
For S3 sources:
- fs.s3a.assumed.role.sts.endpoint = sts.<aws-region>.amazonaws.com
- fs.s3a.endpoint = s3.<aws-region>.amazonaws.com
For Glue sources:
- aws.region = <aws-region>
Prior to adding any sources, an S3 gateway VPC endpoint and an STS interface VPC endpoint must be created in the VPC.
October 11, 2023
Users who were not assigned to the ADMIN
role were unable to run queries against tables and views that did not have an owner. Owners were missing from tables and views created prior to the August 17 update of Dremio Cloud. For some tables and views, an error scenario could have caused the owner to be missing.
October 6, 2023
Users could drop a table or view from an Arctic catalog if they had USAGE
and COMMIT
privileges on the catalog and SELECT
privileges on the table or view. With this update, only users with USAGE
and COMMIT
privileges on the catalog and OWNERSHIP
privileges on a table or view, or users in the ADMIN
role, can drop a table or view.
In the Dremio console, it appeared as though a user without OWNERSHIP
privileges on an Arctic catalog could delete the catalog, even though they could not.
In some cases, queries that used CONVERT_FROM
in a filter condition were failing.
September 21, 2023
Dremio Arctic and all of its related features are no longer in preview mode.
In the Dremio console, ownership in Sonar and Arctic is now listed separately from other privileges, at the top of the Privileges page, and the procedure for transferring ownership is streamlined. For more information, see Transferring Organization Ownership and Transferring Ownership.
Added keyboard shortcuts in the SQL Runner for showing or hiding the Text-to-SQL panel and for triggering Text-to-SQL. For more information, see Keyboard Shortcuts.
The Record Count column has been moved next to the Current Footprint column in the Project Settings > Reflections table.
Added support for the il-central-1: Israel (Tel Aviv) region to the AWS Glue source.
Dremio Cloud provides more helpful information in the error message if an invalid tag or branch name is supplied.
You can now leave the Database Name field blank in a PostgresSQL source.
Reduced the number of S3 lookups required for Arctic DML and DDL operations to improve performance in query planning.
Reduced the amount of heap memory used by the query plan cache.
Updated the Source API to prevent sending secret values in clear text.
Setting query_label()
as an engine rule was resulting in an exception error.
In some cases, default raw reflection matching was not working as expected for users not assigned to the ADMIN
role.
After searching for and selecting a username or role in catalog privileges, the search string was not automatically being cleared.
The Tableau and Power BI buttons were visible in the SQL Runner for unsaved queries.
Saving a query as a view without having run the query was resulting in an error if no engine replicas were active.
Fixed an issue with COL_LIKE()
when input and pattern contained the %
character.
LIKE
was not being highlighted as a reserved keyword in the SQL editor.
Fixed an issue that was causing the use of GRANT ALL
on a project to fail with an "invalid project privilege" error.
OPTIMIZE
and VACUUM
queries on tables with reflections were being evaluated for reflection matching, causing an error.
Fixed an issue with filter pushdowns that was causing some preview queries to fail even though the same query was successful when using Run.
September 12, 2023
Fixed an issue that was causing the RST_STREAM closed stream
error when processing large result sets via JDBC or ODBC.
September 8, 2023
Some queries were failing with a Failure getting source
error.
Queries against views and tables in an Arctic catalog were not showing up on the Jobs page. This issue only affected organizations that use the default Arctic catalog instead of spaces.
September 6, 2023
The following regions are now supported in the AWS Glue source:
- ap-south-2: Asia Pacific (Hyderabad)
- ap-southeast-3: Asia Pacific (Jakarta)
- ap-southeast-4: Asia Pacific (Melbourne)
- eu-south-2: EU (Spain)
- eu-central-2: EU (Zurich)
- me-central-1: Middle East (UAE) DX-69347
Added a new table function, SYS.RECOMMEND_REFLECTIONS
, that recommends aggregation reflections to accelerate existing SQL queries. For more information, see Reflection Recommendations.
This update adds support for the following SQL functions:
- APPROX_PERCENTILE DX-62151
- ARRAY_AVG DX-65324
- ARRAY_CAT DX-67718
- ARRAY_COMPACT DX-67718
- ARRAY_GENERATE_RANGE DX-67718
- ARRAY_MAX DX-65324
- ARRAY_MIN DX-65324
- ARRAY_POSITION DX-67718
- ARRAY_REMOVE DX-65324
- ARRAY_REMOVE_AT DX-67718
- ARRAY_SIZE DX-67718
- ARRAY_SUM DX-65324
- NORMALIZE_STRING DX-68631
The Status of some failed queries was being reported as RUNNING instead of FAILED in the Jobs Overview page.
Updated com.google.guava:guava to 32.1.1-jre to address CVE-2023-2976 in Dremio's internal Iceberg fork.
Updated validation settings to ensure that only privileged users could view Acceleration Settings on the Project Settings > Reflections page.
Fixed an issue that was causing inconsistent query results when ARRAY_CONTAINS
was used with nullability checks.
Plans for queries containing CONVERT_FROM
could not be cached.
To address a CONCURRENT_MODIFICATION
error seen in concurrent metadata refresh queries on Parquet tables, if the query is submitted by scheduler, failures are ignored. If the query is submitted by users, the failed query is retried until it succeeds.
August 28, 2023
When viewing catalog or folder contents on the Datasets page, the "More" menu (...) for tables and views now contains a Delete option, allowing users with appropriate privileges to delete a table or view.
For some failed queries, status in the Job profile was being reported as RUNNING instead of FAILED.
When creating a new project, the Arctic catalog name was not being validated prior to launching the CloudFormation template.
The Open Results link on the Jobs overview page was not working as expected for queries that were run from edit mode on the Dataset page.
When saving a view, the items in the "Save View As" dialog were not sorted in the same way as on the Dataset page.
For organizations created prior to August 17, 2023, granting or revoking table or view privileges for the first time via SQL was successful, but an error was produced on the Organization Settings > Privileges page.
If a dataset name or the name of a parent folder contained a space or ampersand (&) character, clicking on the dataset would populate the SQL Runner with a truncated SELECT
statement.
August 17, 2023
As of this update, each Sonar project in new Dremio Cloud organizations will come deployed with an Arctic catalog, which will support data management capabilities (folders, tables, etc.) for the project. This primary Arctic catalog replaces your home space.
You can now use Role-Based Access Control (RBAC) privileges to control which roles and users can read, write, and manage tables and views in Arctic catalogs.
Dremio Sonar now supports the same SQL syntax as Spark when working with Arctic/Nessie sources.
Dremio automatically optimizes incremental reflection file size to improve reflection performance.
In this update, you must explicitly create folders (namespaces) before creating tables or views in them.
The Usage page under Organization Settings now includes usage data for Arctic catalogs.
Dremio will avoid a full data scan for simple aggregations on partition columns, reading the manifest metadata instead, which improves performance for queries on very large tables.
After DML operations against unpartitioned Iceberg tables, Dremio now compacts the data files written by the DML operation to improve future read performance.
Updated the Snowflake connector to fix intermittent issues when adding Snowflake as a source.
If you optimized a table in Dremio Arctic (Optimize Once) and then viewed the dataset settings for another table, the Optimize Once button remained disabled unless you refreshed the page.
Logged in users were getting redirected to the login page instead of the create password page when clicking on an invite that had not yet been accepted.
When running a job multiple times, the status and job link for the last attempt are now displayed as expected.
Increased concurrency limits to avoid errors when concurrent inserts into the same table were being sent from different streams.
Fixed an issue that could cause a memory leak when querying an Iceberg table with positional deletes.
At times, the DAY()
function was returning either integer or timestamp, depending on how the query was written.
Fixed an issue that was causing an error when running OPTIMIZE TABLE
on a table with reflections.
In some cases, the IF EXISTS
option for DROP BRANCH
and DROP TAG
was being ignored.
Top-level CASE
statements intended to return a Boolean were not being rewritten correctly, resulting in an error for some SQL Server queries.
Some SQL Server queries with nested CASE
statements were failing with invalid SQL comparison syntaxes.
Fixed an issue with the LEFT()
SQL function on Oracle sources for queries with dates.
Some date subtraction queries were not getting pushed down for Oracle sources.
July 27, 2023
July 24, 2023
This update provides performance improvements in the Jobs listing page, and any user with sufficient privileges can now view reflection jobs in the table.
The details panel displaying Wiki content is now available inside a folder on the main branch in an Arctic source.
This update adds support for a new connector that allows querying data from Apache Druid. For more information, see Apache Druid.
You can now drag and drop a table from your home space into the Text-to-SQL panel in the SQL Runner.
We have made some improvements in the efficiency of the queries suggested when using the Text-to-SQL feature.
July 7, 2023
Dremio Cloud has new onboarding flow that walks you through creating your first project and gives you an overview of Dremio features. You can hide the tutorials if you like, and show them again from the Help menu.
June 29, 2023
This update enables new privileges for Arctic catalog-level role-based access control: COMMIT, CREATE BRANCH, CREATE TAG, MANAGE GRANTS, OWNERSHIP, USAGE, and WRITE SETTINGS.
Breaking Change: With this change, non-privileged users (e.g., users who are not catalog or organization owners) will not be able to access Arctic catalogs and their underlying datasets. Catalog or organization owners will need to explicitly grant these privileges to users and roles on the Arctic catalog, as described in Privileges.
The metadata card is now displayed for datasets in Arctic catalogs. For more information, see Viewing the Metadata of an Arctic Dataset.
You can now build reflections on versioned Arctic tables and views. For more information, see Using Arctic Branches with Reflections.
The "Data-as-code" source category has been renamed "Arctic Catalogs."
You can now add the SampleDB source to any project using the Add Data Source dialog.
The "Profile" tab on the Jobs detail page has been renamed "Visual Profile."
You can now add wiki content and labels to Dremio Arctic catalogs that are available in your Sonar project (main branch only).
Autocomplete has been improved to provide better and faster context-aware suggestions for SQL keywords, catalog objects, and functions while you are constructing SQL statements.
When dragging a dataset into the SQL editor, the AT BRANCH | TAG | COMMIT <reference>
will be appended to the query.
Dremio now supports LIST literals using the ARRAY
keyword. For more information, see Data Types.
The schema for the sys.organization.usage
table has been changed to support Arctic catalog usage. The project_id
column has been split into entity_id
and entity_type
so you can quickly determine if usage belongs to a Sonar project or an Arctic catalog. For more information, see SYS.ORGANIZATION.USAGE.
Added validation to ensure that you cannot create an Engine Routing rule with the same name as an existing rule.
The Preview engine cannot be disabled or deleted because it is used to run metadata refresh queries for relational sources and other background activities.
Fixed a security issue that could, in some cases, allow users to bypass email verification for new accounts.
Improved the message that is displayed when an Engine Routing rule rejects a query or if the query does not match any rules.
June 14, 2023
Dremio's Text-to-SQL is now available in Preview! Check out Analyzing Data with Text-to-SQL for an example.
June 7, 2023
You can now use the REST API to cancel a Dremio Arctic optimization job. For more information, see Canceling a Job.
You can now cancel a Dremio Arctic optimization job from the Catalog Settings > Configuration page. For more information, see Canceling a Job.
The Project Settings page now includes an option to manage preferences for the Dremio console. The SQL settings option is no longer available, and the Autocomplete and Copy or Download Results settings are now available under Preferences. For more information, see Dremio Preferences.
You can now configure the default behavior in the Dremio console when clicking on a dataset to either open the SQL Runner with a default SELECT
statement on the dataset or open the Datasets page. For more information, see Querying a Dataset on Click.
The metadata card for datasets now includes links to launch a BI tool. For more information about the metadata card, see Viewing the Metadata of a Dataset.
This update includes changes that provide improved performance for planning times for queries on Dremio Arctic views.
The name and type of a dataset's source is now displayed when viewing the dataset in the lineage graph.
Added more security around DML permission checks to ensure that users cannot access data to which they should not have access.
After modifying and saving the details of a user and selecting another user to edit, the details of the previously modified user were being displayed very briefly in the user detail fields.
Some reflection refreshes that were accelerated by default raw reflections were not being matched in user queries.
Users who had not been assigned to the ADMIN
role could see the Add Space button when viewing all spaces.
In some cases, invalid partition stats were causing the row count estimate for a table to come back as 1.
In the dataset details panel, wiki content was being truncated if it was too long. You can now scroll horizontally on wiki details.
At times, during background metadata refresh, some folders on Amazon S3 sources that had been formatted as tables were losing their formatting. This was causing the PLAN ERROR: UnsupportedOperationException
error for external dashboards that used views or reflections built on those tables.
Empty, disabled reflections were being created if you edited an existing reflection.
When using AWS PrivateLink and handling large amounts of data, system table queries could exceed the default timeout, resulting in a FlightRuntimeException
error.
Optimization jobs were initially being displayed at the bottom of the Jobs list, but they will now be included in the list according to start time.
When promoting some datasets, quoted fields were not being treated as quoted when white space was included before it.
Queries on History tables were failing intermittently.
GRANT
commands on catalog entities were failing with "Role/User \<ID> not found" if existing user or role grantees were no longer present in the system.
In some cases, when clicking the Functions button in the SQL Runner, the list of functions was empty.
When opening a versioned view in a new tab, the Dremio console was displaying an error instead of loading the dataset.
Right-clicking on a dataset and selecting "Open in New Tab" was opening a new browser tab, but the SQL Runner was empty.
June 1, 2023
When you add the first Sonar project to your Dremio Cloud organization, a new sample source called "SampleDB" is now included in the project. To get started with Dremio and learn how to query data across multiple sources and more, check out Getting Started with Dremio Sonar.
May 24, 2023
Right-clicking on a dataset and selecting "Open in New Tab" was opening a new browser tab, but the SQL Runner was empty.
Queries on sys.project.history
tables were failing intermittently.
When clicking a dataset to query it, the "Unsaved Changes" dialog was being displayed in the SQL Runner after changes had already been saved or when no changes had been made.
When using AWS PrivateLink and handling large amounts of data, system table queries could exceed the default timeout, resulting in a FlightRuntimeException
error.
May 9, 2023
The metadata card for datasets now includes a link to the lineage graph. For more information about the metadata card, see Viewing the Metadata of a Dataset.
For Apache Iceberg tables, you can now use the EXPIRE SNAPSHOTS
option in the VACUUM TABLE
SQL command to remove old snapshots and files associated with these snapshots that are no longer needed. For more information, see Expiring Snapshots of Apache Iceberg Tables and VACUUM TABLE.
Dremio Cloud now allows you to rewrite manifest files and optimize select partitions for Apache Iceberg tables:
Manifest files can be rewritten based on a size criteria, enabling the optimization of the metadata, which helps reduce query planning time. For more information, see Optimizing Tables.
You can optimize selected partitions using the partition filter clause. For information about using this SQL command, see OPTIMIZE TABLE.
This update includes two new SQL functions:
ST_GEOHASH
returns the corresponding geohash for given latitude and longitude coordinates (see ST_GEOHASH).ST_FROMGEOHASH
returns the latitude and longitude coordinates of the center of a given geohash (see ST_FROMGEOHASH).
In this update, you can enable or disable the options to download or copy query results from the SQL Runner by navigating to Project Settings > SQL. For more information, see Result Set Actions.
Clicking on a dataset now takes you to the SQL Runner with a SELECT *
query on the dataset. To edit a view, you can click the pencil icon that appears on the far side of the view’s row, to the right of the number of jobs, or in the metadata card.
Dremio was taking a long time to list views in a space and tables in a source, but performance was good when listing the same in the SQL Runner's Data panel.
April 11, 2023
In this update, you can now use an AWS CloudFormation template to add a Sonar project. For more information, see Adding a Project.
You can now rewrite manifest files and optimize select partitions for Iceberg tables in Dremio.
Manifest files can be rewritten based on a size criteria, enabling the optimization of the metadata, which helps reduce query planning time. For more information, see Optimizing Tables.
You can optimize selected partitions using the partition filter clause. For information about using this SQL command, see Optimize Table.
When available, version context is now included with table references in the query profile to indicate more clearly which version of a table is used in a given query.
When viewing a SQL script, the script ID is included in the page URL, allowing you to copy and share the URL with another user in your organization. The user must have VIEW
privileges on the script that you are sharing.
You can now save scripts that contain up to 250,000 characters.
In this update, you can expand or collapse individual phases in the raw profile view.
The Add Project button is no longer displayed on the Sonar Projects page if the logged in user does not have the privilege to create a project.
March 11, 2023
Fixed an issue where duplicate field names in a view could result in a "Some virtual datasets were out of date and have been corrected" error.
March 10, 2023
Arctic-related APIs to manage data optimizations native to Arctic are now available. The APIs available to optimize Arctic tables and views use the catalogs endpoint. These APIs enable you to configure engines, manage schedules, and run optimization jobs. For more information, see Arctic.
Dremio Arctic now provides an automated jobs service that optimizes the storage of Iceberg tables to ensure queries are run efficiently. This jobs automation helps you manage the accumulation of the data files that occurs through DML operations. For more information, see Automatic Table Optimization.
You can now manage organization-level privileges for Dremio Cloud under Organization Settings > Privileges. For more information, see Privileges.
The metadata card for datasets has been updated to indicate whether or not a dataset has reflections, when the dataset was created and last updated, and the dataset owner. For more information, see Viewing the Metadata of a Dataset.
In this update, you can now use an AWS CloudFormation template to add a cloud to your organization. For more information, see Adding a Cloud with an AWS CloudFormation Template.
After signing up for Dremio Test Drive, users are now properly directed to the Datasets page in the Test Drive project instead of to the organization home page.
The commit history for a Dremio Arctic view or table no longer shows entries from other views and tables.
The details page no longer displays a 404 error when you click to view details from the commit history in Dremio Arctic.
Reflections that are configured to never expire will not be dropped.
A view delegation error that prevented queries run by users who are not assigned the ADMIN
role from being accelerated and caused query profiles to contain substitution errors is now fixed.
Users can now save the modified SQL for a table as a view in Dremio Arctic without first running a query.
Updating the SQL for a saved view in Dremio Arctic no longer produces a generic Something went wrong error.
The Format Query option is now consistently available in the SQL editor.
XML responses from AWS Glue are now handled properly and no longer cause queries to fail.