Changelog
We continue to add new features and fix issues to enhance Dremio Cloud. This changelog provides you with a record of those changes.
April 21, 2022
Made some updates to cookie handling for the Tableau login flow to ensure that users are directed to the right page when signing in.
If a user tries to sign in after being removed from an organization, they will automatically be directed to the signup page instead of the login page for the organization.
Starring Catalog Items: This release of Dremio allows you to star frequently used catalog items, such as views, tables, spaces, and sources, which adds them to your Starred list for easier access. The Starred list holds up to 25 entities at a time, and items remain on the list if you open a new browser or clear the cache. See Starring a Dataset for more information.
This release includes various updates to icons and other elements of the Dremio app to enhance usability and overall design.
Multi-line SQL: You can now execute multiple SQL statements in the SQL Editor. Statements run sequentially in the order they appear. Each statement will generate a unique Job ID, and the results of each statement will be displayed in a separate tab in the result area. See Querying Your Data for more information.
SQL Autocomplete: Autocomplete is now available in the SQL Editor. When enabled, autocomplete lets you view and insert possible completions in the editor using the mouse or the arrow keys with Tab or Enter. Autocomplete can provide suggestions for SQL keywords, catalog objects, and functions while you are constructing SQL statements. Suggestions depend on the current context. The autocomplete feature can be enabled or disabled for all users using the Project Settings.
When creating the first project, the wizard was allowing users to proceed without selecting an AWS Region.
The organization name was not being displayed on the login page for Tableau users who were returning to the Dremio app.
Users were not seeing all organizations with which their email address was associated on the organization page, even though all cookies were enabled.
Some requests to create a new engine were failing with a 502 or 500 error.
Microsoft Azure Synapse Analytics was showing up as an available external source, though it is not currently supported.
Fixed an issue that was causing the following error when trying to open a view in the Dataset page: Some virtual datasets are out of date and need to be manually updated.
When viewing All External Sources in the Datasets view, Nessie repositories were included in the list of external sources.
Previously configured storage credentials were not being preserved when editing a Nessie source.
If Power BI and Tableau were disabled, they were still available under the Analyze menu.
Engine replicas were being shut down in the middle of query execution if the corresponding engine was modified or disabled while the query was running.
When there were too many projects and engines to display on a single page under Usage settings, users could not scroll down to see the full list.
Email addresses are now URL encoded when making API calls for Tableau logins to avoid issues with emails containing special characters.
The name of the view currently open in the Dataset page was changing to New Query (edited) after running a filter in the SQL editor.
If Wiki content on the catalog page was too long, the left navigation bar was not resizing as the page was scrolled down.
Fixed an issue where the icon for switching between Light Mode and Dark Mode would not display properly when in Light Mode on Job Details page.
In some cases, not all DCU consumption details were showing up under Billing > Transactions.
During periods of high load, when a large number of concurrent queries were running, users may have seen jobs failing with UNKNOWN or MongoWriteException errors.
Queries that failed due to a recoverable exception, such as a schema change or concurrent metadata refresh for internal refresh queries, are now listed on the Jobs page as Running instead of Failed.
When viewing job details, from the Jobs page or the Run link in the SQL Runner, the status of some jobs was incorrect in the case of multiple query attempts.
ASC and DESC were not being highlighted as reserved keywords in the SQL Editor.
The CREATE TABLE command was not working when used with IF NOT EXISTS.
Sorting by Role or Status on the settings page for Organization > Users is not working as expected and has been disabled.
When viewing the details of a failed query from the Jobs page, if the query had been cancelled because it exceeded the currently defined memory limits, a generic/unknown No value present may pop up with the details page.
If you try to access the Execution tab for a job and the information is unavailable, Dremio will render new content related to the status of the information. If a job has failed, Dremio will report that the information is unavailable. If the job can still be cancelled, Dremio will report that the information is not yet available.
The first query issued on a new data source or running on a newly created engine or preview engine will take slightly longer to run than subsequent queries.
When Arrow caching is enabled, some query runs may show inconsistent results. To avoid this issue you can disable Arrow caching.
On rare occasions, a query can hang for a long time, especially when multiple queries are run while the engine is starting up. To work around this issue, you can run the query again or restart the engine.
A DROP TABLE query may appear to run successfully, even if the table being dropped doesn’t exist.
In some cases, the dataset jobs count in the folder view can be empty, despite the dataset having jobs associated with it. Use the Jobs page to see the accurate list of jobs.
March 21, 2022
Dremio Cloud now offers powerful SQL database-like functionality on Apache Iceberg tables through industry-standard SQL commands. For more information, see SQL Commands for Apache Iceberg Tables.
If you want to change to Dremio’s Standard edition but have Enterprise features configured, those features will be indicated along with links to the configuration pages that control them.
The design (color, border style, etc.) of some UI elements in Dremio have been updated to match the design of the rest of the app.
The peak memory usage shown in the operator profile has been updated to show the maximum of memory reserved and memory used.
The Nessie version used in Dremio Arctic has been upgraded to 0.21.2.
In the Query Visualizer, the Processing Time label has been changed to Runtime, and Memory is now Total Memory.
When clicking Remove on the organization selection form, users were getting directed back to the sign up page.
At times, using the project switcher to change the current project to an Arctic project was generating an unknown error.
Multiple background metadata refresh jobs were getting triggered at the same time for a dataset, causing some jobs to fail.
Fixed an issue that was displaying a brief message about queue assignment rules when navigating to the Engine Routing page from the Reflections page under Project Settings.
Fixed an issue with metadata refresh that could result in incorrect results or query exceptions due to an expected row count mismatch.
If a login method was disabled but that same method was stored in a cookie, users could still use that method to log in.
If a user was active in one or more organizations, they were unable to sign up for a new organization.
In some cases, engines were failing to start in a new Dremio Cloud organization.
In some cases, when viewing usage data for the current month and there was no data available for the current month, users were seeing an unknown error.
Merging in an Arctic project was failing if there were more than 20 commits on the source branch.
The Nessie endpoint was not being passed when a Nessie source was added.
Usage across all engines was not getting aggregated correctly, so total usage data was incorrect.
Users were able to enter custom input when selecting columns to add Measures to a Reflection, but custom input should not have been enabled and was generating an unknown error.
A table cannot be selected as the destination for saving a view, but users were able to select a table when saving a view, and the view was not saved.
When a user was deleted, the notification confirming the action was showing the deleted user as undefined instead of as the actual user ID.
When users tried to add a tag to a dataset’s catalog, a Tags are not saved error was produced even though the tag was still applied.
A query with not in was not returning the correct results for Hadoop and Hive datasets.
Clicking Discard in the SQL Runner was not clearing the query contents.
Some queries against ORC tables in a Glue source were failing with the following error: Hadoop native library is required for Hive ORC data, but is not loaded
March 7, 2022
March 1, 2022
A link to full terms of the invoice/billing agreement is now available in the Add Billing Account dialog.
Various updates have been made to improve query execution as well as general app performance.
When editing an engine, the Last Replica Auto-Stop field under Advanced Configuration requires a minimum value of 60 seconds.
The SQL Runner now allows you to save your SQL as a script. See Querying Your Data for more information.
Changed a generic internal server error message that was displaying when trying to cancel a query preview that had finished. The message now indicates that the completed job cannot be cancelled.
Standard Edition users can no longer create roles, enterprise identity providers, or external token providers, and Dremio now provides tooltips to indicate that these are Enterprise features.
Disabled the Delete All button under Account Settings > Personal Access Tokens if there are no tokens.
Engine scaling events (adding or removing replicas) are now captured in the Dremio Cloud audit log.
We’ve made some changes to the way engine status is displayed in the SQL Runner to make job and engine state easier to understand.
Users can now expand (|→) or collapse (←|) the catalog explorer panel when working in the SQL Editor or SQL Runner.
Added the ability to push down DATE_ADD and DATE_SUB functions to RDBMS sources.
Added the ability to push down DATE_ADD and DATE_SUB functions to RDBMS sources.
The queue_name column has been removed from the jobs runtime system table (SYS.PROJECT.JOBS) because queues are not supported in Dremio Cloud.
In SYS.PROJECT.VIEWS and SYS.PROJECT."TABLES", if an owner of a view or table has been deleted, owner_id now shows up as $deleted$ instead of the invalid user ID.
The state of a project created using the /projects API will now return as either CREATING or ACTIVE.
The requestId attribute is now optional when creating a project with the /projects API.
The following ruleSet attributes have been renamed in the Engine Rules (/rules) API: ruleName -> name, ruleCondition -> condition, ruleAction -> action, outputValue -> engineName, PLACE -> ROUTE (action attribute). Additionally, the following attributes have been removed, outputId, ruleUniqueId, and version.
Fixed an issue that was causing errors when running the CloudFormation template during the automatic cloud connect portion of the signup process.
During signup, user input in the Organization and Project fields was not getting validated, resulting in a generic error when trying to submit.
If password authentication was disabled after a user had started the signup process from the invite link or by entering their email address on the login page, the user could still log in with a password.
Fixed an issue that was redirecting users to the login screen when trying to remove an organization from the organization selection screen.
If a user was active in one or more organizations, they were unable to sign up for a new organization.
Continuing the sign up process on an incomplete organization was resulting in a CloudFormation template error.
The Reflections tab was not showing up under Project Settings for local or PUBLIC users.
Tooltips to provide details about engine states were not always displaying when expected.
If a user’s login session had expired and they tried to visit a page in the app, an unexpected error was showing up instead of taking the user directly to the login page.
The tooltip to indicate the status of an engine that was starting up was not displaying unless users hovered over the engine dropdown.
When browsing the Dremio app with Edge, scrollbars were not showing up and users had to zoom out to view certain portions of the screen.
Not all users were showing up when searching in the User filter on the Jobs page.
The Upload File button on the Datasets page was available all the time, regardless of whether or not it had been disabled.
The Tableau and PowerBI buttons were not showing up or remaining hidden as expected, and they are now enabled all the time in the SQL Editor.
If password authentication was disabled after a user had started the signup process from the invite link or by entering their email address on the login page, the user could still log in with a password.
When adding a tag to a dataset’s catalog, a Tags are not saved error was displayed even though the tag was still applied.
After removing a view from a space, the total number of views in the space was not getting updated in the Spaces panel.
In some cases, if a Parquet file in a Delta Lake table had many row groups, COUNT(*) queries were failing due to a divide by 0 exception.
Even though ownership of a folder or table had been granted to a role, users belonging to that role were unable to access the folder or table.
Ownership (owner_id in SYS.ORGANIZATION.USERS) of invited users was not being set to the ID of the user who invited them.
IN filters were not working after applying an internal schema on a varchar file.
Long paths to a dataset were running into the tab names above the SQL Editor.
Fixed an issue that was causing some reflections to fail with a NessieReferenceConflictException error.
The Billing transactions API was not filtering results properly for a billing account.
February 15, 2022
Audit logging is now available in Dremio Cloud. See Audit Logging and SYS.PROJECT.HISTORY.EVENTS for more information.
Email verification is now required during user sign up when using username/password authentication.
If a user had not completed the onboarding process during sign up, the incomplete organization was showing up on the login page and there was no way to remove the incomplete organization. The option to remove such organizations is now available.
We’ve simplified and streamlined our login flow by using cookies. If a user has accepted the use of all cookies, the organization selection and login screens will be skipped if the user still has an active login session. Additionally, the main login screen will be skipped if the user has logged in previously from the current browser (even if the login session has expired).
Dremio Cloud Admins now have access to Billing, where they can change between Dremio Standard or Enterprise editions, create and modify a billing account, and view detailed transaction data for DCU usage.
Apache Iceberg is now a supported format for tables.
Copyright information is now displayed directly in the help menu, and the About Dremio option has been removed. Edition details can now be found in the Billing section under Organization Settings.
Each OAuth application listed under Organization Settings now includes the application ID.
IP addresses are now used instead of hostnames to track replicas, meaning that DNS hostnames no longer need to be enabled in the VPC. Additionally, the Engines detail page uses Host IP instead of Hostname and lists replica IP addresses.
Our enhanced CloudFormation Template provides updated IAM policies with narrowed permission scopes to enhance customer security.
The is_member condition for building Engine Rules was not working for internal roles.
The support portal link on the Help menu is only available to Admin users in organizations using the Enterprise edition of Dremio Cloud.
We’ve added new engine states to provide more granular details about engine status. For more information, see Managing Engines.
Users with the PUBLIC role can now view the OAuth Application page under Organization Settings.
Admins can grant or revoke the CONFIGURE SECURITY privilege to a user or role. Users who have been granted this privilege (directly or via role) will have full access to Authentication, External Token Providers, BI Applications, and OAuth Applications under Organization Settings.
Runtime system tables for tables and views are now available in Dremio Cloud. See SYS.PROJECT.“TABLES” and SYS.PROJECT.VIEWS for more information.
Runtime and historical jobs tables are now accessible as sys.project.jobs and sys.project.history.jobs.
An engine is no longer created when adding a project with the /projects API. Users can create engines independent of projects.
Clicking the help link in the Add OAuth Application dialog was resulting in a 404/page not found error.
Fixed a Job <job-id> output doesn’t exist error message to provide more information about why the job could not be retrieved and how the user can address the issue.
Fixed an issue that was causing an unknown error when searching on the Engines page.
Fixed an issue that was preventing users from disabling SSO for Power BI by simply clearing the Enable checkbox and saving — the Save button was not available in this scenario.
If a non-admin user had ownership granted for a view, the view was not showing up for the user.
We’ve fixed an issue that would occasionally result in lost connectivity to a source.
When adding a new project, the Cloud used for the latest project is selected by default.
When hovering over a specific entry in the Usage graph, Dremio now shows details that include the date and DCUs used on that day.
When reordering an engine on the Engine Routing page, both an error and a success message were being displayed.
When viewing usage details for a specific project, changing the usage period was taking users back to the All Projects view.
Users were not able to modify any engine attributes via the Engines API unless they specified all available attributes. All attributes are optional now.
November 2021
The bug that prevented you from assigning the Administrator role to users has been resolved.
The logos for social login options have been fixed to correctly match with the text.
Errors messages for issues with granting and revoking privileges and not having priviliges to perform an action have been enhanced to be specific and clear.
Enhanced error message to include parsing message when the engine rule being added is syntactically incorrect (there was an UNKNOWN error being returned previously).
October 2021
Added support for the eu-west-2 (London) region.
Fixed a deadlock with processing engine scaling events, which prevented engines from scaling up and shutting down.
In addition to admin users, users with the OWNERSHIP or MANAGE GRANTS privileges on an object can now transfer ownership to a new user.
Improved error messages when a user attempts to run ALTER USER with a PASSWORD clause (this action is not supported through SQL).
You can now successfully add a user as an admin on the Administrators page.
September 2021
EU Control Plane: We now support the eu-central-1 (Frankfurt) and eu-west-1 (Ireland) AWS regions for the control plane, allowing for better latency and performance with EU-based data planes. For additional information on EU endpoints, see the Supported Regions page.
The PUBLIC role no longer has privileges to modify, disable, or delete engines.
Added a Default Project setting, which tools like Tableau and Power BI default to so you don’t have to specify a project ID to connect to Dremio.
EU Control Plane: The API endpoint for the control plane is https://api.eu.dremio.cloud.
Personal access tokens are supported in the authorization header.
EU Control Plane: The App endpoint for the control plane is https://app.eu.dremio.cloud.
The login page of https://app.dremio.cloud displayed the deleted organizations. Now, the login page only displays the active organizations.
When an engine was created with a duplicate name through a POST operation, a 500 Internal Server Error response code was returned. The error handling has been corrected to return a 400 Bad Request response code when an engine is created with a duplicate name.
Previously deleted organizations were displayed on the login page. This has been fixed so that only active organizations are listed on the log in page.
The Documentation link in the Help menu now points to https://docs-beta.dremio.com.
On the SQL Editor page, the search tab in the Datasets pop out has been removed because it the search function doesn’t currently work. It will be added back when the search issues have been resolved.
Querying sys.privileges to retrieve the granted privileges did not return privileges on the current organization and the current project. Organization and project-level privileges and the associated organization and project are now returned correctly when querying sys.privileges.