16.0.0 Release Notes (Dremio April 2021)

What’s New

Access Control

Dremio 16.0.0 provides administrators a more extensive range of privileges to control user and group access to Dremio objects. Role-based access control is governed by object inheritance and the scope applied when granting privileges to a Dremio user. See Role-based Access for more information.

Contact Dremio Technical Support for assistance in enabling this feature.

Restoring Dremio

The new access control functionality in Dremio v16.0 is considered a breaking change. We recommend implementing the new access control functionality on a lower environment, such as staging or test environments, prior to installing this update on a live instance. This is to help identify any problems that might arise within the Dremio environment when the new access control functionality takes effect, and potentially mitigates any damage to your data on a live environment.

Restoring Dremio

If you run into issues with upgrading the KVStore, run the command ./dremio-admin repair-acls -d.

Slicing Thread Monitor

In instances when a slicing thread (execution thread) receives significant traffic that causes it to hang or slow, this monitor flags the thread and either creates a new thread or activates a thread from the free thread pool. Existing tasks in the slower thread’s queue are then redirected to the newly-activated thread. The original thread is left to complete the current task fragment and then exits the system.

Idle Time Connection Settings

Created new idle timeout limit and connection number options for RDBMS data sources. When the established connection limit and/or time is reached the connection is automatically killed. Idling is controlled using the Maximum idle connections and Connection idle time (s) options under the Advanced Options tab of an Relational Database source’s settings.

Content Security Policy Metadata Tag

Added a Content Security Policy (CSP) metadata value that lists all domains through which frontend resources, such as Javascript, CSS, and images, are allowed to be loaded in Dremio.

Large Expression Support

Dremio has been optimized to handle case statements containing more than 200 case expressions. New internal logical expression type treat these case expressions as flat branches instead of nested, which means upward of 800 branches are now supported.

This functionality is not enabled by default. If you wish to enable this functionality, contact Dremio’s Professional Services team.

Plan Statistics

Generating table statistics has now been added to Dremio, which allows for improved optmization of query execution plans. Via the sys.table_statistics system table, this functionality introduces ANALYZE TABLE commands for the computing and deletion of statistics for physical datasets (PDS). Dremio may now identify the following statistics: estimated number of distinct values, number of rows, and number of null values.

After computing statistics, the results are accessed via Dremio’s Query Planner by enabling the Planner.use_statistics option.

Encrypted LDAP Passwords

Dremio 16.0.0 enhances security by providing support for storing encrypted the LDAP bindPassword in the Dremio keystore. See Encrypting the LDAP Bind Password and the dremio-admin encrypt command for more information.

Updated Kubernetes Helm Chart

Dremio 16.0.0 features an updated Helm chart for Kubernetes-based deployments.

Restoring Dremio

For Kubernetes users, Dremio v16.0.0 introduces a breaking change which requires an update to your Helm chart. Please see the following page for the specific change necessary to your Helm chart. For enterprise customers with customized Helm charts that require additional assistance, please contact Dremio Support.

New Gandiva Functions

Dremio 16.0.0 provides the following new Gandiva functions:

  • Trigonometric: sin, cos, asin, acos, tan, atan, sinh, cosh, tanh, cotg, radians, degrees
  • Date/Time:last_day
  • Miscellaneous: sha1 and sha256
  • Conversion: CONVERT_FROM(expression, 'UTF8', replacement char)
  • Cast: castTIME(timestamp) and castVARCHAR(numeric_types)

Change in return type of SQL last_day( ) Function

Please note that in Dremio v.16.0.0, the return type of the SQL last_day( ) function has been changed to DATE, to align with the standard and expected behavior of this function. In prior versions of Dremio, this function returned a date-formatted STRING value, which caused unexpected behavior when passed to other functions expecting DATE input parameters.

Please note, for customers who have already modified their queries to account for the previous last_day() return type of STRING, they may see some query failures with this new behavior. The solution is to wrap the call to last_day() in a call to to_char(), as follows:

select to_char(last_day(current_date), 'YYYY-MM-DD') as last_day_string

Preview: Google Cloud Storage (GCS) Connector

Google Cloud Storage has been added as an option for file system sources, like S3 and HDFS. To enable and test out this new functionality, contact Dremio Customer Support for assistance.

Changed Behavior

By default, Dremio 16.0.0 uses the Tableau SDK Connector rather than the ODBC driver to export data in Tableau.

Other Enhancements

Dremio 16.0.0 supports runtime filtering on non-partitioned columms.

Dremio 16.0.0 no longer loads SSL-related classes when SSL is disabled.

Dremio 16.0.0 improves performance for Dremio clients by allocating memory only once for Netty.

Fixed Issues in 16.0.0

**Dremio can’t display usernames with double quotation marks and colons**
Fixed by disallowing double quotation marks and colons in usernames.

Unable to promote Parquet-formatted files without .parquet extension in AWS Edition
Resolved by using a format matcher during the promotion flow for Parquet-formatted files.

CVE-2017-15288: Vulnerability reported against scala-2.10.1.jars
Resolved by removing the vulnerable library from Dremio distributions and updating the scala library to 2.10.7.

Known Issues

The MODIFY and ALTER SQL statements have the following known limitations with regard to Access Control:

  • A user may lose access to a dataset by executing a MODIFY SQL statement that impacts metadata.
  • A user that does not have access to the parent object cannot promote a dataset using the ALTER SQL statement.