18.0.0 Release Notes (Dremio August 2021)

Upgrading to 18.0

We recommend that you follow this guide before upgrading Dremio as v18.0 introduces multiple features that have a significant impact on the performance of your queries. This guide offers suggestions for how best to verify the functionality of each new feature while still protecting your data.

What’s New

Local Users & Roles

Additional functionality has been added to access control with regard to the creation and management of users and roles:

  • Both internal and external users are supported in the same instance of Dremio. In addition, new user management screens allow for administrators to add multiple users by username as well as assign roles and view permissions on a user-by-user basis.
  • Roles, or groups of privileges and sub-roles with users assigned, allow you to control what privileges are inherited by a user based on their position or capacity.

Additional APIs as well as SQL commands have been added to extend this functionality.

New Job History & Job Details Screens

The Jobs History and Details screens in Dremio have been updated both from a usability and layout standpoint. The original dual-paned screen listing jobs on the left and job details on the right is now a full-screen interactive experience.

While jobs are still listed in a standard table fashion with individual rows for each query and its details separated into columns, the information presented here is now expanded through the inclusion of multiple new columns as well as an interactive hover capability.

Clicking on an individual job will now take you to a screen full of data, such as a breakdown of the query execution, scans, dataset graphs of all views and datasets used, and more.

Note

A support key must be used to enable this new functionality. However, in a future release of Dremio, the new UI will be enabled by default.

Query Visualizer

As part of the new Job Details screen, the Query Plan Visualizer is a new visual component that allows users to view a graphical representation of the SQL execution tree. With this tool, all resources used for the query are identified, including nodes, reflections, and individual datasets. Those nodes deemed most expensive, meaning time-intensive, will appear more colorful, whereas the less-used resources will appear dimmed.

Reflection Refresh Job Routing (Preview)

Reflection refreshes may now be assigned to specific queues via the ALTER TABLE SQL command. From the SQL Editor, you may now set all reflections in a dataset to an individual or default workload queue. Adding reflections to queues is still dependent on the Workload Management functionality and queues being set up for managing concurrent queries.

To enable this functionality and use it in your current environment, please contact Dremio Customer Support.

Other Enhancements

Near-Real-Time Metadata Refresh

Metadata refreshes now occur in near real time for almost any dataset size. This is accomplished through the adoption using executors and reading data changes incrementally for almost-instantaneous refreshes.

For more information on how to enable this functionality, see Near-Real-Time Metadata Refreshes. In a future version of Dremio, this will be automatically enabled by default.

Note

This functionality will be enabled by default in a future release of Dremio.

Near-Real-Time Metadata Refresh for Reflections

Metadata refreshes for reflections are now able to take place in near real time for any dataset size. This is accomplished through the use of incremental scans of data partitions rather than full scans of the reflection’s metadata for each refresh.

For more information on how to enable this functionality, see Near-Real-Time Metadata Refreshes for Reflections. In a future version of Dremio, this will be automatically enabled by default.

Note

This functionality will be enabled by default in a future release of Dremio.

Unlimited Splits for Hive & FileSystem Sources

Due to the changes made to allow for near-real-time metadata refresh rates, Dremio’s limitation on the total number of splits for FileSystem sources is removed.

For more information on how to enable this functionality, see Unlimited Splits.

Note

This functionality will be enabled by default in a future release of Dremio.

Deprecations

Mixed Type Removal

Dremio has removed support for columns with mixed data types for FileSystem sources. A standard schema is now enforced. Columns with compatible mixed data types will be promoted to a single data type. However, columns with incompatible mixed data types will be unreadable by Dremio until such instances are resolved. Visit this page for additional information.

If you wish to test whether your datasets will be impacted by this change, please contact Dremio Support or your Field Technician prior to upgrading.

User/Role v2 APIs

All v2 user (/api/v2/user/*), group (/api/v2/group/*), and role (/api/v2/role/*) API functionality will be deprecated in Dremio v19.0 and removed. We recommend that you begin transitioning to /v3/ APIs for users, groups, and roles.

Fixed Issues

Dremio could not push down queries to Oracle due to an unsupported regex expression.
An unsupported regex expression using \Q\E is now not pushed down to Oracle when it is present, allowing Dremio to handle the regex.

When attempting to run a query with a constant key in GROUP BY for SQL Server, the query failed.
Dremio automatically rewrites the query before the physical plan is created by removing constant keys from GROUP BY, thereby avoiding the error.

Dremio had no support for adding S3 connection properties when asynchronous access was enabled._
Dremio now offers the ability to configure the number of connections for S3 as well as each S3 connection property, including the following options:

  • fs.s3a.connection.maximum - S3 Connection pool size.
  • fs.s3a.connection.establish.timeout - S3 Connection time out.
  • fs.s3a.connection.timeout - S3 Socket timeout.

Known Bugs

On Mapr sources, when unlimited splits is enabled users are encountering failures with the CREATE_TABLE privilege. This will be fixed in Dremio v18.1.

Notice About Dremio 16.0+ Upgrades

If prior to 16.0, a source granted privileges to specific users for CREATE TABLE or DROP, when upgrading to 16.0+ Dremio will grant the PUBLIC role these privileges, rather than the original users. This behavior is incorrect, and we recommend that after upgrading to Dremio 16.0+, administrators should check any sources with CREATE TABLE and DROP privileges to ensure that proper access is granted.