Changelog
We continue to add new features and fix issues to enhance Dremio Cloud. This changelog provides you with a record of those changes.
June 1, 2023
When you add the first Sonar project to your Dremio Cloud organization, a new sample source called “SampleDB” is now included in the project. To get started with Dremio and learn how to query data across multiple sources and more, check out Getting Started with Dremio Sonar.
Users with the organization-level MANAGE GRANTS privilege who have not been assigned the ADMIN role are not able to assign privileges to users or roles unless they have been explicitly assigned the CREATE USER or CREATE ROLE privilege. This issue will be fixed in a future release so that users with MANAGE GRANTS for the organization can assign privileges without needing ADMIN or CREATE USER/CREATE ROLE.
At times, some folders on Amazon S3 sources that have been formatted as tables can lose their formatting. This can cause the PLAN ERROR: UnsupportedOperationException error for external dashboards that use views or reflections built on those tables. Until this issue is fixed, you can reformat the folder and recreate the reflections that are used in the dashboard.
A new external ID is generated every time a cloud entity is edited, which breaks the IAM role trust relationship. To prevent this issue, do not edit a cloud entity when you are using IAM role credentials. If you face this issue, temporarily switch to IAM access key credentials with the same permissions as the IAM role.
May 24, 2023
Right-clicking on a dataset and selecting “Open in New Tab” was opening a new browser tab, but the SQL Runner was empty.
Queries on sys.project.history tables were failing intermittently.
When clicking a dataset to query it, the “Unsaved Changes” dialog was being displayed in the SQL Runner after changes had already been saved or when no changes had been made.
When using AWS PrivateLink and handling large amounts of data, system table queries could exceed the default timeout, resulting in a FlightRuntimeException error.
May 9, 2023
The metadata card for datasets now includes a link to the lineage graph. For more information about the metadata card, see Viewing the Metadata of a Dataset.
For Apache Iceberg tables, you can now use the EXPIRE SNAPSHOTS option in the VACUUM TABLE SQL command to remove old snapshots and files associated with these snapshots that are no longer needed. For more information, see Expiring Snapshots of Apache Iceberg Tables and VACUUM TABLE.
Dremio Cloud now allows you to rewrite manifest files and optimize select partitions for Apache Iceberg tables:
-
Manifest files can be rewritten based on a size criteria, enabling the optimization of the metadata, which helps reduce query planning time. For more information, see Optimizing Tables.
-
You can optimize selected partitions using the partition filter clause. For information about using this SQL command, see OPTIMIZE TABLE.
This release includes two new SQL functions:
-
ST_GEOHASHreturns the corresponding geohash for given latitude and longitude coordinates (see ST_GEOHASH). -
ST_FROMGEOHASHreturns the latitude and longitude coordinates of the center of a given geohash (see ST_FROMGEOHASH).
In this Dremio Cloud release, you can enable or disable the options to download or copy query results from the SQL Runner by navigating to Project Settings > SQL. For more information, see Result Set Actions.
Clicking on a dataset now takes you to the SQL Runner with a SELECT * query on the dataset. To edit a view, you can click the pencil icon that appears on the far side of the view’s row, to the right of the number of jobs, or in the metadata card.
Dremio was taking a long time to list views in a space and tables in a source, but performance was good when listing the same in the SQL Runner’s Data panel.
April 11, 2023
In this release, you can now use an AWS CloudFormation template to add a Sonar project. For more information, see Adding a Project.
You can now rewrite manifest files and optimize select partitions for Iceberg tables in Dremio.
-
Manifest files can be rewritten based on a size criteria, enabling the optimization of the metadata, which helps reduce query planning time. For more information, see Optimizing Tables.
-
You can optimize selected partitions using the partition filter clause. For information about using this SQL command, see Optimize Table.
When available, version context is now included with table references in the query profile to indicate more clearly which version of a table is used in a given query.
When viewing a SQL script, the script ID is included in the page URL, allowing you to copy and share the URL with another user in your organization. The user must have VIEW privileges on the script that you are sharing.
You can now save scripts that contain up to 250,000 characters.
In this release, you can expand or collapse individual phases in the raw profile view.
The Add Project button is no longer displayed on the Sonar Projects page if the logged in user does not have the privilege to create a project.
March 11, 2023
Fixed an issue where duplicate field names in a view could result in a “Some virtual datasets were out of date and have been corrected” error.
March 10, 2023
Arctic-related APIs to manage data optimizations native to Arctic are now available. The APIs available to optimize Arctic tables and views use the catalogs endpoint. These APIs enable you to configure engines, manage schedules, and run optimization jobs. For more information, see Arctic.
Dremio Arctic now provides an automated jobs service that optimizes the storage of Iceberg tables to ensure queries are run efficiently. This jobs automation helps you manage the accumulation of the data files that occurs through DML operations. For more information, see Automatic Table Optimization.
You can now manage organization-level privileges for Dremio Cloud under Organization Settings > Privileges. For more information, see Privileges.
The metadata card for datasets has been updated to indicate whether or not a dataset has reflections, when the dataset was created and last updated, and the dataset owner. For more information, see Viewing the Metadata of a Dataset.
In this release, you can now use an AWS CloudFormation template to add a cloud to your organization. For more information, see Adding a Cloud with an AWS CloudFormation Template.
After signing up for Dremio Test Drive, users were getting directed to the organization home page instead of the Datasets page in the Test Drive project.
The commit history for a Dremio Arctic view or table was showing entries from other views and tables.
At times, when clicking for details from the commit history in Dremio Arctic, the details page was displaying a 404 error.
Some reflections were getting dropped even though they were configured to never expire.
Fixed an issue with view delegation that was causing queries run by users not assigned the ADMIN role to not be accelerated, and query profiles contained substitution errors.
Fixed an issue that was preventing users from saving the modified SQL for a table as a view in Dremio Arctic without first running the query.
Trying to update the SQL for a saved view in Dremio Arctic was producing a generic Something went wrong error.
On occasion, the Format Query option was unavailable in the SQL editor.
In some cases, XML responses from AWS Glue were not being handled properly and causing queries to fail.
February 17, 2023
Fixed an issue that was causing problems with AWS PrivateLink connectivity to the Dremio Cloud control plane.
February 7, 2023
In this release, when executing queries in the SQL Runner, Dremio displays the states of the running job in the query results area until the query results are displayed.
At times, when trying to add an Arctic catalog using the name of a catalog that had been previously deleted, Dremio was producing an error and the catalog was not created.
The Sources and Scripts lists on the SQL Runner page were not scrollable when too many items were available.
Query results were not being displayed after opening and running a versioned view.
Fixed an issue that was causing the same heading to be displayed for each column section in the Review changes dialog after modifying a view configured with a reflection and clicking the Show more link on the Reflections tab.
When viewing a table or view on the Datasets page, the Catalog tab has been renamed Details.
When running queries against an Arctic source using the Arrow Flight SQL JDBC driver, the queries were failing with a Tenant context should not be null error.
In some cases, queries against text tables with evolved schemas were failing with an ArrayIndexOutOfBoundsException error.
January 20, 2023
Some queries using multiple CONVERT_FROM functions on different JSON data type columns were failing to read with an Unable to find the referenced field error.
January 19, 2023
This release includes a new SQL function, COL_LIKE, which tests whether an expression column matches a pattern column. For more information, see COL_LIKE.
In some cases, listing Arctic datasets from Sonar would result in a generic Something went wrong error.
At times, when trying to add an Arctic catalog using the name of a catalog that had been previously deleted, Dremio was producing an error and the catalog was not created.
When running queries against an Arctic source using the Arrow Flight SQL JDBC driver, the queries were failing with a Tenant context should not be null error.
January 10, 2023
You can now view or modify dataset details without having to go to the Datasets page and click on the Details tab. Wherever the icon for a dataset is displayed in Dremio, you can hover over the icon to display the metadata card, which now displays label information as well as a link to Open Details. The details link opens the metadata details panel on the right side of the browser, letting you add or remove labels and view or edit the Wiki associated with the dataset. For more information, see Viewing the Metadata of a Dataset.
In this release, a new source connector allows you to query data from a Dremio software cluster. For more information, see Connecting to a Dremio Software Cluster.
This release adds support for a new connector that allows querying data from Snowflake data warehouses. For more information, see Snowflake.
This release supports the use of BROADCAST hints in queries to distribute data across all execution engines. Hints must be entered as /*+ <hint> */, which is standard across data warehouses. Prior to this release, text enclosed in /* */ was treated as as a comment. Dremio will continue to treat text enclosed in /* */ as a comment unless the first character is +. For more information, see Distributing Data Evenly Across Executor Nodes During Joins.
You can now hide the SQL editor if you want to focus the page on the results table. If you hide the editor and navigate to a different page, the editor will remain hidden when you return to the SQL runner. For more information, see SQL Editor.
In the SQL editor, you can now format your SQL using the Format SQL shortcut (Cmd + Shift + f or Ctrl + Shift + f). As long as the current syntax is valid, the SQL formatter applies a conventional style to your query by aligning commands for readability. For more information, see SQL Editor.
In some cases, clicking on a specific role for details under Organization Settings > Roles was generating a 404/page not found error.
If you entered a non-matching value when filtering for users on the Jobs page, the contents of the User filter was empty the next time it was opened.
In some cases, with the Arrow Flight SQL ODBC driver, users were getting an error when testing the connection to Microsoft Excel in the ODBC Administrator on Windows.
Details about access times for some catalog operations were not included in query profiles.
Some queries executed against an AWS Glue source were failing because the source appeared to be unavailable.
When opening a reflection to view details under Settings > Reflections, an error indicating that the reflection did not exist could be displayed, even though the reflection was valid.
The cursor and selected text were misaligned in the SQL Runner in some browsers running on Windows.
Fixed an issue that was causing queries failed if certain expression splits contained CAST AS UNION.
In the SQL runner, some of the actions text under query results was not displaying correctly when using Safari.
When using search to filter all scripts in the SQL runner, clearing the contents of the search box was generating an unexpected error.
Made some improvements in approximation of double value to nearest decimal when filtering on a table with double type, but files contain decimal values.
Previous Releases
December 21, 2022
Fixed an issue that was causing misalignment of the cursor and selected text in the SQL Runner in some browsers running on Windows.
December 19, 2022
Fixed an issue that was causing the non-highlighted portions of a query or commented lines in a query to be cleared from the SQL editor when running or previewing the query.
Details about access times for some catalog operations were not included in query profiles.
December 14, 2022
You can now optimize Iceberg tables to maximize the speed and efficiency of data retrieval. Rewrite data files using a compaction process to combine small files into larger files or split large files to reduce metadata overhead and runtime file open costs. For more information, see Optimizing tables.
You can now roll back to a previous state of an Iceberg table using either a snapshot ID or a timestamp reference. For more information, see Rolling back tables.
This release includes a new SQL function, ARRAY_CONTAINS, which returns whether a list contains a given value. For more information, see ARRAY_CONTAINS.
The LIKE operator can now take optional keywords ANY, ALL,or SOME to allow matching against a list of patterns. For more information, see LIKE.
Breaking Change: If you mix implicit and explicit joins, only the last of the implicitly joined tables can be in the ON clause. Otherwise you will receive a “Table not found” error. For example,
select *
from
NAS2."customer.parquet" c,
NAS2."nation.parquet" n
left join
NAS2."orders.parquet" as o
on c.c_custkey = o.o_custkey
results in the error Table 'c' not found. The solution is to replace the comma with an explicit cross join like this:
select *
from
NAS2."customer.parquet" c
cross join
NAS2."nation.parquet" n
left join
NAS2."orders.parquet" as o
on c.c_custkey = o.o_custkey
This is functionally equivalent since implicit joins implement a cross product of the two tables.
Added a copy button to all instances of a Commit ID in Arctic Catalogs.
In Dremio Arctic, all Commit IDs have been updated to be 8 characters long.
The button to add an Arctic Catalog to a Sonar project was not visible if an organization did not contain any projects.
When trying to use an invalid name for a branch in Dremio Arctic, details are now provided about allowed characters and supported naming conventions.
The author of a commit in Dremio Arctic was being displayed as the author’s internal ID instead of his or her username.
For some users, when trying to log in to Dremio Cloud using Okta, the login would fail and return users to the login prompt.
When loading some views in BI tools to create a visualization, more fields were being displayed than actual columns in the view. If any of the excess fields were used in the visualization, an error was produced.
For tables created from a folder of files, the jobs count on the Datasets page was incorrect as it always showed 0.
Improved the error message that is displayed when trying to create a new folder with a duplicate name in the same location.
If you clicked on a job to view details, your position on the page was reset when clicking the Back button or the Jobs link on the page header. Your position on the main Jobs page is now maintained in these scenarios.
In some cases, with the Arrow Flight SQL ODBC driver, users were getting an error when testing the connection to Microsoft Excel in the ODBC Administrator on Windows.
Clicking Discard in the SQL Runner would not clear the query context if you hadn’t first run a query.
When copying a view definition and pasting into the SQL editor, the pasted SQL was incorrect because newlines were not being retained.
Fixed an issue that was causing accelerated queries to fail after a project was reactivated or restored.
Removed duplicate user query alternative plans during reflection matching to reduce logical planning times.
Fixed an issue that was causing COALESCE queries containing NULLIF calls to not get pushed down to Oracle.
November 30, 2022
If a query contained CONVERT_FROM() on a large json literal string, the query was failing with an OutOfMemoryException error.
Running SHOW SCHEMAS was failing in when used with Arrow Flight, causing failures with Apache Superset.
Fixed an issue that was causing some queries to fail with a NullPointerException in com.dremio.exec.store.iceberg.DremioInputFile.newStream().
November 17, 2022
This release includes performance improvements for incremental metadata refreshes on partitioned Parquet tables.
This release contains a number of fixes to address potential security issues.
Fixed an issue that was causing a tooltip to fail to appear when trying to execute a query using a stopped engine.
Navigating to a page URL with an invalid project ID was failing to produce a 404, as expected.
When working in an Arctic Catalog source, the branch was getting reset to main if you clicked the History button to go to the Commits page.
For tables created from a folder of files, the jobs count on the Datasets page was incorrect as it always showed 0.
Fixed an issue that was causing the status of a cancelled job to show as RUNNING or PLANNING.
If you clicked on a job to view details, your position on the page was reset when clicking the Back button or the Jobs link on the page header. Your position on the main Jobs page is now maintained in these scenarios.
When a materialization took too long to deserialize, the job updating the materialization cache entry could hang and block all reflection refreshes.
INFORMATION_SCHEMA.COLUMNS queries on Arctic Catalogs were failing if Dremio was unable to access the Iceberg metadata file.
Fixed a bug that was creating a race condition, causing REFRESH REFLECTION and REFRESH DATASET jobs to hang when reading Iceberg metadata.
Fixed an issue that was causing Oracle pushdowns to fail on JOIN queries that used the TO_DATE function.
November 7, 2022
For Dremio Arctic (Preview), you can now add the Arctic catalog as a source within either the Arctic service or the Sonar service. For more information, see Connecting an Arctic Source to Sonar.
For Dremio Arctic (Preview) catalogs that are available in your Sonar project, you can now access tables and views that are on the main branch. For more information, see Creating a Live Connection to a Dataset from Dremio Cloud.
November 4, 2022
Nested views based on tags in Arctic were not resolving properly.
When working in an Arctic source, if the branch name included a space in its name, Dremio was producing a generic “Unexpected error occurred” when creating a branch.
If you created an empty folder in an Arctic source, you couldn’t delete it using the Remove Folder action.
Fixed an issue that was taking users from a page under Settings to an invalid page in the SQL Runner.
If you selected some text in the Dremio app, the Copy Selection popup was appearing in the upper-right corner of the screen instead of next to the selected text.
On the Jobs page, you can now right-click on a job’s row and use a new popup to view the job details, open the job details in another tab, or copy the job ID.
On catalog entities, ownership granted to a role was not being inherited by users in that role.
Fields requiring paths to an S3 bucket are now pre-populated with the s3:// prefix, so you only have to include the remainder of the path.
When a materialization took too long to deserialize, the job updating the materialization cache entry could hang and block all reflection refreshes.
In the SQL Runner, if you expanded an object in the Data panel while the object was empty, new items created in the object would not show up until you refreshed the page.
If you highlighted a single statement in the SQL Runner and used the keyboard shortcut to run it, Dremio was running the entire query instead of only what had been highlighted.
The starting_ts and starting_ts_epoch columns have been added to sys.project.history.jobs table. These columns contain the date/time and time since epoch when the job entered the STARTING state.
October 21, 2022
Fixed an issue that was preventing any users added after the first user from seeing or querying the Dremio sample dataset in the first Sonar project.
October 19, 2022
We have continued to streamline and simplify the sign-up and onboarding flow. All you need to do is choose your authentication method and provide a name for your organization. After Dremio finishes creating all of the required cloud resources and your first project, you’ll end up on the new landing page, where you can manage organization settings or dive deeper into Dremio Sonar or Dremio Arctic.
Dremio Cloud has a new home page for your organization where you can quickly access organization settings, Sonar Projects, and Arctic Catalogs. For more information, see Navigating Dremio Cloud.
Dremio Arctic (preview) now supports the Arctic Catalog, an Iceberg catalog that enables you to list and manage Iceberg tables and views. An Arctic catalog consists of one or more branches, and a branch contains zero or more folders, tables and views. For more information, see Getting Started with Arctic, Overview of Dremio Arctic, and Arctic Catalog.
By default, your first project now contains the Dremio sample data source, with the NYC Taxi trip data promoted to a table and ready for your first query. From the organization landing page, click Add Sonar Project on the Sonar card to add your first project, then go to Getting Started with Dremio Sonar to see how quickly you can run your first query.
To help you better manage cloud costs, Dremio Cloud’s preview engine is now configured to stay disabled until you need it. Regular tasks like metadata refresh will run less often by default, and those jobs are bundled to consolidate execution time.
NOTE: If the preview engine is not running and you want to add a new database source, Dremio will ask you to confirm that you want to start the preview engine before proceeding.
Dremio now supports wide tables. See Datasets for more information and limitations.
This release includes a number of fixes that resolve potential security issues.
Updated the PostgreSQL JDBC Driver to version 42.4.1 to address CVE-2022-31197.
Updated org.eclipse.jetty:jetty-io to version 9.4.47 to address potential security issues.
When provisioning users, groups, and roles from Azure AD to Dremio Cloud, roles were not being assigned to added users. The issue was caused by a case mismatch in SCIM API calls. To address this issue in Dremio Cloud, SCIM APIs are now more lenient regarding the case-sensitivity of JSON keys and sub-types.
Frequent, consecutive requests to the Job API endpoint to retrieve a Job’s status could result in an UNKNOWN StatusRuntimeException error.
At times, when using Dremio’s Azure AD integration, users would see threads blocked on a call to Dremio’s internal Azure AD cache.
Disabling the preview engine was causing connection issues for database sources.
Fixed an issue that was causing REFRESH REFLECTION and REFRESH DATASET jobs to hang when reading Iceberg metadata.
Under certain conditions, such as during Dremio Cloud maintenance windows, reflection jobs or dataset metadata jobs may stop refreshing.
When a materialization took too long to deserialize, the job updating the materialization cache entry could hang and block all reflection refreshes.
Some queries were failing with a message indicating an error that could not be retried, caused by a sort operation that exceeded the maximum amount of memory.
BRANCH was not getting highlighted as a pre-reserved keyword in the SQL editor.
For some queries on views using OR or IN, unnecessary subquery scans were not removed, causing inconsistent results.
On catalog entities, ownership granted to a role was not being inherited by users in that role.
Fixed an issue that was causing the error GandivaException: Failed to make LLVM module due to Function double abs(double) not supported yet for certain case expressions used as input arguments.
On occasion, projecting complex data types would result in a Schema change exception.
October 18, 2022
When provisioning users, groups, and roles from Azure AD to Dremio Cloud, roles were not being assigned to added users. The issue was caused by a case mismatch in SCIM API calls. To address this issue, SCIM APIs are now more lenient regarding the case-sensitivity of JSON keys and sub-types.
September 28, 2022
To help avoid unnecessary costs, Dremio automatically deactivates any project that has not been accessed in the last 15 days. Dremio sends a courtesy email to project owners three days prior to deactivation. In addition, users with OWNERSHIP privileges or users in the ADMIN role can archive a project. Inactive and archived projects do not consume any EC2 resources and do not incur any costs in your AWS account. See Managing Projects for more information.
Fixed an issue that was causing REFRESH REFLECTION and REFRESH DATASET jobs to hang when reading Iceberg metadata.
Under certain conditions, such as during Dremio Cloud maintenance windows, reflection jobs or dataset metadata jobs could stop refreshing.
September 19, 2022
When trying to sign up for Dremio Cloud using the Google, Microsoft, or GitHub options, the page was stuck in a loading state for several minutes and eventually closed with an error.
September 8, 2022
We’ve made a number of changes to continue our ongoing UI improvements (e.g., better tooltips, improved error messages, etc.).
Fixed an issue that could cause the Dremio app to hang when changing the current Dremio edition.
In some cases, frequent scale up and scale down events were occurring on the Preview engine, and more replicas than the maximum were in use.
When a dataset was using too many tags to display in a single line on the Datasets page, there was no way to expand the list to see all tags. In these cases, Dremio now displays an ellipsis button (…) to expand the list of tags.
If the Preview engine was stopped (i.e., minimum replicas set to zero) and you clicked on a view from the Datasets page, no preview was displayed but Dremio started the Preview engine. The screen was blocked by a popup dialog indicating that the Preview engine was starting.
Tags and Wiki content were being hidden for any promoted dataset. In this release they are only hidden for datasets belonging to Data-as-code sources.
In some cases, if users had saved a large number of SQL Runner scripts and tried to save another, Dremio could take several seconds to create the script and refresh the page.
August 22, 2022
We’ve made a number of changes to continue our ongoing UI improvements (e.g., better tooltips, improved error messages, etc.).
Changes made to the columns displayed on the Jobs page, or the order of the columns, were not being saved after leaving the page.
For some queries, when viewing the details of a job that was opened from the Jobs page, clicking the Open Results link in the upper-right corner of the page resulted in a dataset tmp.UNTITLED version ... error on the Datasets page.
When looking at the query Graph for a view and clicking the query icon of the parent table, the Dataset Settings from the results were showing settings for the view instead of the table.
In rare cases, an issue in the planning phase could result in the same query returning different results depending on the query context.
After changing the engine configuration, some queries were failing with an IndexOutOfBoundsException error.
When a query that used a reflection was executed multiple times, some of the jobs used the reflection and some did not.
Dremio was failing to parse queries on a view when the query originated from a JDBC/ODBC client that had the quoting connection property set to a non-default value.
Some queries that used the FLATTEN() function were showing results for a Preview, but no data was returned when using Run.
In some scenarios, invalid metadata about partition statistics was leading to inaccurate rowcount estimates for tables. The result was slower than expected query execution or out of memory issues. For each table included in a query where this behavior appears, perform an ALTER TABLE <table-name> FORGET METADATA, then re-promote the resulting file or folder to a new table. This will ensure that the table is created with the correct partition statistics.
VIEW was not being highlighted as a pre-reserved keyword in the SQL editor.
August 9, 2022
When signing up for Dremio Cloud and creating a new organization, the CloudFormation template now supports AWS Key Management Service for encrypting the project store. After launching the template in the final step of the sign-up wizard (Step 3 of 3 - Configure AWS resources), you can select any of the following for Encryption Type: SSE-S3 (default), SSE-KMS (AWS Managed Key), SSE-KMS (Customer Managed Key)
Note:
If you select SSE-KMS (AWS Managed Key), AWS will create and manage the key for you. If you select SSE-KMS (Customer Managed Key), you will need to provide the KMS Key ARN of the key you have already created.
For Dremio Arctic (Preview), we’ve added support for views, enabling you to create, replace, show, and drop views in your data source. Views created in Arctic have the notion of versions and are Iceberg views (see 0.14 Iceberg View Spec and the Project Nessie GitHub repo for the experimental implementation). Also see the limitations with this release of Arctic views.
In addition to sharing with users, you can now share scripts saved in the SQL Runner with a role (see Adding Privileges).
Fixed an issue that was causing failures in Microsoft SQL Server queries that contained a boolean filter set to true.
INSERT, MERGE, UPDATE, TRUNCATE, and DELETE queries in the SQL Runner were failing with an Invalid path error when using a partial key/path.
CONVERT_FROM queries were returning errors if they included an argument that was an empty binary string. This issue has been fixed, and such queries have been optimized for memory utilization.
July 21, 2022
We’ve made additional enhancements to simplify the signup flow, such as removing unrelated or unknown organizations and ensuring that you return to the correct organization when resuming sign-up.
If you close the cookie consent or cookie preferences dialog without accepting functional cookies, we’ll disable functional cookies for you and won’t show the cookie consent dialog again.
We’ve made some improvements to correctly validate inputs when you add a project and select the IAM Role option for Project Data Credentials.
Based on feedback from our users, Dremio no longer creates an engine named First Engine when signing up.
Dremio now supports reflections on the sys.project.history.jobs and sys.project.history.events system tables.
Changes to Cloud and Edition are now included in the audit log, and the events are available in the default project for an organization.
When a job has completed successfully and doesn’t have a profile, the Profile page now contains more information about why the profile/execution details are not available.
June 29, 2022
Based on feedback from our customers, we have made several enhancements to the sign-up experience. We have added more helpful information and additional details about various options and fields that need to be filled in, provided more feedback about progress, and new users are now logged in directly to their new organization when sign-up is complete.
When you open a dataset in the SQL editor, Dremio will no longer generate a preview of the dataset by default. You will need to click Preview or Run to view the dataset.
June 16, 2022
Added a close button to the cookie consent dialog on the Dremio Cloud and Tableau login pages.
Dremio now supports row-access and column-masking policies for row and column controls over user query access to sensitive tables, views, and columns. This allows administrators to dynamically exclude or mask private data at the column and row levels prior to query execution and without physically altering the original values.
Extra spaces will now be trimmed from fields to avoid errors when adding external sources.
Added a plus button to the upper-right corner of the page for spaces and Nessie sources that allows users to quickly upload a file or add a new folder, table, or view.
The option to enable Arrow caching in advanced reflection settings has been removed because Arrow caching is not supported with unlimited splits.
This release adds to existing Iceberg DML capabilities allows users to run DELETE, UPDATE, MERGE, and TRUNCATE statements against Iceberg tables. See SQL Commands for Apache Iceberg Tables for more information.
You can now create SQL scalar user-defined functions (UDFs), which are callable routines that make it easier for you to write and reuse SQL logic across queries. UDFs let you extend the capabilities of Dremio SQL, provide a layer of abstraction to simplify query construction, encapsulate business logic, and support row and column policies for access control.
Added support for internal schema using SQL commands, which lets the user override the data type of a column instead of using the type that Dremio automatically detected.
The DATEDIFF and ADD_MONTHS Hive functions are supported in queries.
Script owners in the SQL Runner are now indicated with a small orange flag next their username. Script owners cannot be removed or have their privileges changed.
The option to save a script will be disabled if the user already has 100 scripts, which is the maximum per user, and the Dremio app now provides a tooltip to provide information why an additional script cannot be saved.
When adding or modifying SQL Runner script privileges, the View privilege is enabled automatically if any of the other privileges are enabled.
The created_by column in the sys.organization.roles table was showing LOCAL for users created via the SCIM API.
If you tried to access the Execution tab for a job that had failed or was in progress, an unknown “Profile Fragment is Empty” error would be displayed. Now Dremio provides more useful information about the job status.
The metadata_retrieval_ts, engine_start_ts, metadata_retrieval_epoch, and engine_start_epoch fields in the sys.project.history.jobs table were empty.
The created_by column was empty for predefined roles (admin, local) in the sys.organization.roles table.
When creating a personal access token, the value in the Lifetime field was displayed as NaN when using the up and down arrows to change the value.
Tree items on the Join page were expanding one level too far, to the column level.
Fixed an issue that was preventing users from scrolling to items at the bottom of the Data panel in the SQL Runner if items near the top had been expanded.
A DROP TABLE query may have appeared to run successfully, even if the table being dropped didn’t exist.
If a query ended with a comment followed by whitespace, the comment was being treated as a query.
When query results were truncated to optimize performance, the notification about truncated results was showing up inconsistently.
May 24, 2022
This release adds support for Okta’s custom authorization server issuer URL: https://${yourOktaDomain}/oauth2/${authorizationServerId}. Previously, only https://${yourOktaDomain} was supported.
The Columns button on the Jobs page has been replaced with the Manage Columns button, and you can change the order of displayed columns using drag and drop.
Dremio now displays the status of replica scale down events as Pending, Completed, or Failed to more accurately represent replica state.
The option to save a script will be disabled if you already have 100 scripts, which is the maximum per user, and the Dremio app now provides a tooltip to provide information why an additional script cannot be saved.
In this release, Dremio supports changing the partition spec of an Iceberg table via the ALTER command.
Along with ROW and ARRAY keywords, you can now use STRUCT and LIST keywords to represent complex data types.
In this release, you can share scripts with others in your organization by adding users and assigning privileges to View, Modify, Manage Grants, or Delete. For more information, see Adding Privileges.
The default save options for an untitled script are Save Script and Save Script as. When editing an active script, the Save Script option is only available if you have MODIFY privileges.
Fixed an issue that was causing users to be directed to the Users page when clicking on the email signup link.
After signing up, Dremio was directing users to the login page instead of taking them to their organization.
In some cases, the Google, Microsoft, and GitHub options were not working as expected during signup.
When Arrow caching was enabled, some query runs may have shown inconsistent results.
When viewing the details of a failed query from the Jobs page, if the query had been cancelled because it exceeded the currently defined memory limits, a generic/unknown No value present error could pop up with the details page.
Fixed an issue that was preventing users from downloading profiles.
An unknown error was being generated when attempting to remove a reflection from the Acceleration dialog and saving the change, and the error would continue to be displayed.
The Default CTAS Format setting was being displayed under General settings for a source instead of Advanced Options.
On rare occasions, a query could hang for a long time, especially when multiple queries were run while the engine was starting up.
The first query issued on a new data source or running on a newly created engine or preview engine was taking slightly longer to run than subsequent queries.
Comma separators had been enabled in some fields where they should not have been enabled, such as Port.
The results table in the SQL Editor was displaying multiple space characters in the query as single spaces in the results.
If a query ended with a comment followed by whitespace, the comment was being treated as a query.
Trying to add a field to a single-line query ending in a semicolon was resulting in a Failure parsing the query error when applying the change.
The option to cancel a running query was not available in the SQL Runner.
April 21, 2022
Made some updates to cookie handling for the Tableau login flow to ensure that users are directed to the right page when signing in.
If a user tries to sign in after being removed from an organization, they will automatically be directed to the signup page instead of the login page for the organization.
Starring Catalog Items: This release of Dremio allows you to star frequently used catalog items, such as views, tables, spaces, and sources, which adds them to your Starred list for easier access. The Starred list holds up to 25 entities at a time, and items remain on the list if you open a new browser or clear the cache. See Starring a Dataset for more information.
This release includes various updates to icons and other elements of the Dremio app to enhance usability and overall design.
Multi-line SQL: You can now execute multiple SQL statements in the SQL Editor. Statements run sequentially in the order they appear. Each statement will generate a unique Job ID, and the results of each statement will be displayed in a separate tab in the result area. See Querying Your Data for more information.
SQL Autocomplete: Autocomplete is now available in the SQL Editor. When enabled, autocomplete lets you view and insert possible completions in the editor using the mouse or the arrow keys with Tab or Enter. Autocomplete can provide suggestions for SQL keywords, catalog objects, and functions while you are constructing SQL statements. Suggestions depend on the current context. The autocomplete feature can be enabled or disabled for all users using the Project Settings.
When creating the first project, the wizard was allowing users to proceed without selecting an AWS Region.
The organization name was not being displayed on the login page for Tableau users who were returning to the Dremio app.
Users were not seeing all organizations with which their email address was associated on the organization page, even though all cookies were enabled.
Email addresses are now URL encoded when making API calls for Tableau logins to avoid issues with emails containing special characters.
Some requests to create a new engine were failing with a 502 or 500 error.
Microsoft Azure Synapse Analytics was showing up as an available external source, though it is not currently supported.
Fixed an issue that was causing the following error when trying to open a view in the Dataset page: Some virtual datasets are out of date and need to be manually updated.
When viewing All External Sources in the Datasets view, Nessie repositories were included in the list of external sources.
Previously configured storage credentials were not being preserved when editing a Nessie source.
If Power BI and Tableau were disabled, they were still available under the Analyze menu.
Engine replicas were being shut down in the middle of query execution if the corresponding engine was modified or disabled while the query was running.
When there were too many projects and engines to display on a single page under Usage settings, users could not scroll down to see the full list.
The name of the view currently open in the Dataset page was changing to New Query (edited) after running a filter in the SQL editor.
If Wiki content on the catalog page was too long, the left navigation bar was not resizing as the page was scrolled down.
Fixed an issue where the icon for switching between Light Mode and Dark Mode would not display properly when in Light Mode on Job Details page.
In some cases, not all DCU consumption details were showing up under Billing > Transactions.
During periods of high load, when a large number of concurrent queries were running, users may have seen jobs failing with UNKNOWN or MongoWriteException errors.
Queries that failed due to a recoverable exception, such as a schema change or concurrent metadata refresh for internal refresh queries, are now listed on the Jobs page as Running instead of Failed.
When viewing job details, from the Jobs page or the Run link in the SQL Runner, the status of some jobs was incorrect in the case of multiple query attempts.
ASC and DESC were not being highlighted as reserved keywords in the SQL Editor.
The CREATE TABLE command was not working when used with IF NOT EXISTS.
March 21, 2022
Dremio Cloud now offers powerful SQL database-like functionality on Apache Iceberg tables through industry-standard SQL commands. For more information, see SQL Commands for Apache Iceberg Tables.
If you want to change to Dremio’s Standard edition but have Enterprise features configured, those features will be indicated along with links to the configuration pages that control them.
The design (color, border style, etc.) of some UI elements in Dremio have been updated to match the design of the rest of the app.
The peak memory usage shown in the operator profile has been updated to show the maximum of memory reserved and memory used.
The Nessie version used in Dremio Arctic has been upgraded to 0.21.2.
In the Query Visualizer, the Processing Time label has been changed to Runtime, and Memory is now Total Memory.
When clicking Remove on the organization selection form, users were getting directed back to the signup page.
At times, using the project switcher to change the current project to an Arctic project was generating an unknown error.
Multiple background metadata refresh jobs were getting triggered at the same time for a dataset, causing some jobs to fail.
Fixed an issue that was displaying a brief message about queue assignment rules when navigating to the Engine Routing page from the Reflections page under Project Settings.
Fixed an issue with metadata refresh that could result in incorrect results or query exceptions due to an expected row count mismatch.
If a login method was disabled but that same method was stored in a cookie, users could still use that method to log in.
If a user was active in one or more organizations, they were unable to sign up for a new organization.
In some cases, engines were failing to start in a new Dremio Cloud organization.
In some cases, when viewing usage data for the current month and there was no data available for the current month, users were seeing an unknown error.
Merging in an Arctic project was failing if there were more than 20 commits on the source branch.
The Nessie endpoint was not being passed when a Nessie source was added.
Usage across all engines was not getting aggregated correctly, so total usage data was incorrect.
Users were able to enter custom input when selecting columns to add Measures to a Reflection, but custom input should not have been enabled and was generating an unknown error.
A table cannot be selected as the destination for saving a view, but users were able to select a table when saving a view, and the view was not saved.
When a user was deleted, the notification confirming the action was showing the deleted user as undefined instead of as the actual user ID.
When users tried to add a tag to a dataset’s catalog, a Tags are not saved error was produced even though the tag was still applied.
A query with not in was not returning the correct results for Hadoop and Hive datasets.
Clicking Discard in the SQL Runner was not clearing the query contents.
Some queries against ORC tables in a Glue source were failing with the following error: Hadoop native library is required for Hive ORC data, but is not loaded
March 7, 2022
March 1, 2022
A link to full terms of the invoice/billing agreement is now available in the Add Billing Account dialog.
Various updates have been made to improve query execution as well as general app performance.
The state of a project created using the /projects API will now return as either CREATING or ACTIVE.
The requestId attribute is now optional when creating a project with the /projects API.
The following ruleSet attributes have been renamed in the Engine Rules (/rules) API: ruleName -> name, ruleCondition -> condition, ruleAction -> action, outputValue -> engineName, PLACE -> ROUTE (action attribute). Additionally, the following attributes have been removed, outputId, ruleUniqueId, and version.
When editing an engine, the Last Replica Auto-Stop field under Advanced Configuration requires a minimum value of 60 seconds.
The SQL Runner now allows you to save your SQL as a script. See Querying Your Data for more information.
Changed a generic internal server error message that was displaying when trying to cancel a query preview that had finished. The message now indicates that the completed job cannot be cancelled.
Standard Edition users can no longer create roles, enterprise identity providers, or external token providers, and Dremio now provides tooltips to indicate that these are Enterprise features.
Disabled the Delete All button under Account Settings > Personal Access Tokens if there are no tokens.
Engine scaling events (adding or removing replicas) are now captured in the Dremio Cloud audit log.
We’ve made some changes to the way engine status is displayed in the SQL Runner to make job and engine state easier to understand.
Users can now expand (|→) or collapse (←|) the catalog explorer panel when working in the SQL Editor or SQL Runner.
Added the ability to push down DATE_ADD and DATE_SUB functions to RDBMS sources.
Added the ability to push down DATE_ADD and DATE_SUB functions to RDBMS sources.
The queue_name column has been removed from the jobs runtime system table (SYS.PROJECT.JOBS) because queues are not supported in Dremio Cloud.
In SYS.PROJECT.VIEWS and SYS.PROJECT."TABLES", if an owner of a view or table has been deleted, owner_id now shows up as $deleted$ instead of the invalid user ID.
Fixed an issue that was causing errors when running the CloudFormation template during the automatic cloud connect portion of the signup process.
During signup, user input in the Organization and Project fields was not getting validated, resulting in a generic error when trying to submit.
If password authentication was disabled after a user had started the signup process from the invite link or by entering their email address on the login page, the user could still log in with a password.
Fixed an issue that was redirecting users to the login screen when trying to remove an organization from the organization selection screen.
If a user was active in one or more organizations, they were unable to sign up for a new organization.
Continuing the signup process on an incomplete organization was resulting in a CloudFormation template error.
The Billing transactions API was not filtering results properly for a billing account.
The Reflections tab was not showing up under Project Settings for local or PUBLIC users.
Tooltips to provide details about engine states were not always displaying when expected.
If a user’s login session had expired and they tried to visit a page in the app, an unexpected error was showing up instead of taking the user directly to the login page.
The tooltip to indicate the status of an engine that was starting up was not displaying unless users hovered over the engine dropdown.
When browsing the Dremio app with Edge, scrollbars were not showing up and users had to zoom out to view certain portions of the screen.
Not all users were showing up when searching in the User filter on the Jobs page.
The Upload File button on the Datasets page was available all the time, regardless of whether or not it had been disabled.
The Tableau and PowerBI buttons were not showing up or remaining hidden as expected, and they are now enabled all the time in the SQL Editor.
If password authentication was disabled after a user had started the signup process from the invite link or by entering their email address on the login page, the user could still log in with a password.
When adding a tag to a dataset’s catalog, a Tags are not saved error was displayed even though the tag was still applied.
After removing a view from a space, the total number of views in the space was not getting updated in the Spaces panel.
In some cases, if a Parquet file in a Delta Lake table had many row groups, COUNT(*) queries were failing due to a divide by 0 exception.
Even though ownership of a folder or table had been granted to a role, users belonging to that role were unable to access the folder or table.
Ownership (owner_id in SYS.ORGANIZATION.USERS) of invited users was not being set to the ID of the user who invited them.
IN filters were not working after applying an internal schema on a varchar file.
Long paths to a dataset were running into the tab names above the SQL Editor.
Fixed an issue that was causing some reflections to fail with a NessieReferenceConflictException error.
February 15, 2022
Audit logging is now available in Dremio Cloud. See Audit Logging and SYS.PROJECT.HISTORY.EVENTS for more information.
Email verification is now required during user signup when using username/password authentication.
If a user had not completed the onboarding process during signup, the incomplete organization was showing up on the login page and there was no way to remove the incomplete organization. The option to remove such organizations is now available.
We’ve simplified and streamlined our login flow by using cookies. If a user has accepted the use of all cookies, the organization selection and login screens will be skipped if the user still has an active login session. Additionally, the main login screen will be skipped if the user has logged in previously from the current browser (even if the login session has expired).
An engine is no longer created when adding a project with the /projects API. Users can create engines independent of projects.
Dremio Cloud Admins now have access to Billing, where they can change between Dremio Standard or Enterprise editions, create and modify a billing account, and view detailed transaction data for DCU usage.
Apache Iceberg is now a supported format for tables.
Copyright information is now displayed directly in the help menu, and the About Dremio option has been removed. Edition details can now be found in the Billing section under Organization Settings.
Each OAuth application listed under Organization Settings now includes the application ID.
IP addresses are now used instead of hostnames to track replicas, meaning that DNS hostnames no longer need to be enabled in the VPC. Additionally, the Engines detail page uses Host IP instead of Hostname and lists replica IP addresses.
Our enhanced CloudFormation Template provides updated IAM policies with narrowed permission scopes to enhance customer security.
The is_member condition for building Engine Rules was not working for internal roles.
The support portal link on the Help menu is only available to Admin users in organizations using the Enterprise edition of Dremio Cloud.
We’ve added new engine states to provide more granular details about engine status. For more information, see Managing Engines.
Users with the PUBLIC role can now view the OAuth Application page under Organization Settings.
Admins can grant or revoke the CONFIGURE SECURITY privilege to a user or role. Users who have been granted this privilege (directly or via role) will have full access to Authentication, External Token Providers, BI Applications, and OAuth Applications under Organization Settings.
Runtime system tables for tables and views are now available in Dremio Cloud. See SYS.PROJECT.“TABLES” and SYS.PROJECT.VIEWS for more information.
Runtime and historical jobs tables are now accessible as sys.project.jobs and sys.project.history.jobs.
Users were not able to modify any engine attributes via the Engines API unless they specified all available attributes. All attributes are optional now.
Clicking the help link in the Add OAuth Application dialog was resulting in a 404/page not found error.
Fixed a Job <job-id> output doesn’t exist error message to provide more information about why the job could not be retrieved and how the user can address the issue.
Fixed an issue that was causing an unknown error when searching on the Engines page.
Fixed an issue that was preventing users from disabling SSO for Power BI by simply clearing the Enable checkbox and saving — the Save button was not available in this scenario.
If a non-admin user had ownership granted for a view, the view was not showing up for the user.
We’ve fixed an issue that would occasionally result in lost connectivity to a source.
When adding a new project, the Cloud used for the latest project is selected by default.
When hovering over a specific entry in the Usage graph, Dremio now shows details that include the date and DCUs used on that day.
When reordering an engine on the Engine Routing page, both an error and a success message were being displayed.
When viewing usage details for a specific project, changing the usage period was taking users back to the All Projects view.
November 2021
The bug that prevented you from assigning the Administrator role to users has been resolved.
The logos for social login options have been fixed to correctly match with the text.
Errors messages for issues with granting and revoking privileges and not having priviliges to perform an action have been enhanced to be specific and clear.
Enhanced error message to include parsing message when the engine rule being added is syntactically incorrect (there was an UNKNOWN error being returned previously).
October 2021
Added support for the eu-west-2 (London) region.
Fixed a deadlock with processing engine scaling events, which prevented engines from scaling up and shutting down.
In addition to admin users, users with the OWNERSHIP or MANAGE GRANTS privileges on an object can now transfer ownership to a new user.
Improved error messages when a user attempts to run ALTER USER with a PASSWORD clause (this action is not supported through SQL).
You can now successfully add a user as an admin on the Administrators page.
September 2021
EU Control Plane: We now support the eu-central-1 (Frankfurt) and eu-west-1 (Ireland) AWS regions for the control plane, allowing for better latency and performance with EU-based data planes. For additional information on EU endpoints, see the Supported Regions page.
The PUBLIC role no longer has privileges to modify, disable, or delete engines.
Added a Default Project setting, which tools like Tableau and Power BI default to so you don’t have to specify a project ID to connect to Dremio.
EU Control Plane: The API endpoint for the control plane is https://api.eu.dremio.cloud.
Personal access tokens are supported in the authorization header.
EU Control Plane: The App endpoint for the control plane is https://app.eu.dremio.cloud.
The login page of https://app.dremio.cloud displayed the deleted organizations. Now, the login page only displays the active organizations.
When an engine was created with a duplicate name through a POST operation, a 500 Internal Server Error response code was returned. The error handling has been corrected to return a 400 Bad Request response code when an engine is created with a duplicate name.
Previously deleted organizations were displayed on the login page. This has been fixed so that only active organizations are listed on the log in page.
The Documentation link in the Help menu now points to https://docs-beta.dremio.com.
On the SQL Editor page, the search tab in the Datasets pop out has been removed because it the search function doesn’t currently work. It will be added back when the search issues have been resolved.
Querying sys.privileges to retrieve the granted privileges did not return privileges on the current organization and the current project. Organization and project-level privileges and the associated organization and project are now returned correctly when querying sys.privileges.