Dremio 16.0.0 provides administrators a more extensive range of privileges to control user and group access to Dremio objects. Role-based access control is governed by object inheritance and the scope applied when granting privileges to a Dremio user. See Role-based Access for more information.
Contact Dremio Technical Support for assistance in enabling this feature.
The new access control functionality in Dremio v16.0 is considered a breaking change. We recommend implementing the new access control functionality on a lower environment, such as staging or test environments, prior to installing this update on a live instance. This is to help identify any problems that might arise within the Dremio environment when the new access control functionality takes effect, and potentially mitigates any damage to your data on a live environment.
If you run into issues with upgrading the KVStore, run the command
./dremio-admin repair-acls -d.
If prior to 16.0, a source granted privileges to specific users for
CREATE TABLE or
DROP, when upgrading to 16.0+ Dremio will grant the
PUBLIC role these privileges. This behavior is incorrect, and we recommend that after upgrading to 16.0+, administrators should check any sources with
CREATE TABLE and
DROP privileges to ensure that proper access is granted.
In instances when a slicing thread (execution thread) receives significant traffic that causes it to hang or slow, this monitor flags the thread and either creates a new thread or activates a thread from the free thread pool. Existing tasks in the slower thread’s queue are then redirected to the newly-activated thread. The original thread is left to complete the current task fragment and then exits the system.
Created new idle timeout limit and connection number options for RDBMS data sources. When the established connection limit and/or time is reached the connection is automatically killed. Idling is controlled using the Maximum idle connections and Connection idle time (s) options under the Advanced Options tab of an Relational Database source’s settings.
Dremio has been optimized to handle case statements containing more than 200 case expressions. New internal logical expression type treat these case expressions as flat branches instead of nested, which means upward of 800 branches are now supported.
This functionality is not enabled by default. If you wish to enable this functionality, contact Dremio’s Professional Services team.
Generating table statistics has now been added to Dremio, which allows for improved optmization of query execution plans. Via the
sys.table_statistics system table, this functionality introduces
ANALYZE TABLE commands for the computing and deletion of statistics for physical datasets (PDS). Dremio may now identify the following statistics: estimated number of distinct values, number of rows, and number of null values.
After computing statistics, the results are accessed via Dremio’s Query Planner by enabling the
Dremio 16.0.0 enhances security by providing support for storing encrypted the LDAP
bindPassword in the Dremio keystore. See Encrypting the LDAP Bind Password and the dremio-admin encrypt command for more information.
Dremio 16.0.0 features an updated Helm chart for Kubernetes-based deployments.
For Kubernetes users, Dremio v16.0.0 introduces a breaking change which requires an update to your Helm chart. Please see the following page for the specific change necessary to your Helm chart. For enterprise customers with customized Helm charts that require additional assistance, please contact Dremio Support.
Dremio 16.0.0 provides the following new Gandiva functions:
CONVERT_FROM(expression, 'UTF8', replacement char)
Please note that in Dremio v.16.0.0, the return type of the SQL last_day( ) function has been changed to DATE, to align with the standard and expected behavior of this function. In prior versions of Dremio, this function returned a date-formatted STRING value, which caused unexpected behavior when passed to other functions expecting DATE input parameters.
Please note, for customers who have already modified their queries to account for the previous last_day() return type of STRING, they may see some query failures with this new behavior. The solution is to wrap the call to last_day() in a call to to_char(), as follows:
select to_char(last_day(current_date), 'YYYY-MM-DD') as last_day_string
Google Cloud Storage has been added as an option for file system sources, like S3 and HDFS. To enable and test out this new functionality, contact Dremio Customer Support for assistance.
By default, Dremio 16.0.0 uses the Tableau SDK Connector rather than the ODBC driver to export data in Tableau.
Dremio 16.0.0 supports runtime filtering on non-partitioned columms.
Dremio 16.0.0 no longer loads SSL-related classes when SSL is disabled.
Dremio 16.0.0 improves performance for Dremio clients by allocating memory only once for Netty.
**Dremio can’t display usernames with double quotation marks and colons** Fixed by disallowing double quotation marks and colons in usernames.
Unable to promote Parquet-formatted files without .parquet extension in AWS Edition
Resolved by using a format matcher during the promotion flow for Parquet-formatted files.
CVE-2017-15288: Vulnerability reported against scala-2.10.1.jars
Resolved by removing the vulnerable library from Dremio distributions and updating the scala library to 2.10.7.
ALTER SQL statements have the following known limitations with regard to Access Control:
MODIFYSQL statement that impacts metadata.
Dremio 16.1.0 introduces the ability to disable inline (query-time) checking of metadata validity (expiry) on a per-source basis, which will prevent Dremio from refreshing dataset metadata during query planning.
This new functionality is enabled by first setting a support key; once this key is set, a new checkbox is shown in the Source Configuration UI (Advanced Options) to permit disabling of the query-time metadata validity check.
Metadata Validity Check is disabled for a source:
Refresh Everyinterval in the source configuration). However, the expiry time (
Expire Aftersetting) will not be checked at query time to determine if metadata is stale.
ALTER PDS … REFRESH METADATASQL commands should be issued for these critical datasets to guarantee freshness.
Risk associated with using this option:
There are two steps required to use this option:
store.plugin.show_metadata_validity_checkbox, switch this option on, and Save. See Support Keys for instructions on setting support keys.
Advanced Optionspanel in source configuration. Note: If this option does not appear after the support key is set, refresh your browser window.
The first few queries run after a Dremio restart might appear to experience delays associated with Metadata Retrieval, even after the above option is enabled. However this is due to reloading the Permissions Cache and should only appear just after a restart. To validate the delay is not due to inline refresh after setting the above option, consult the query profile and verify that the time taken for CACHED_METADATA is 0ms.
Dremio 16.1.0 allows Delta Lake to be enabled by default. Dremio administrators may enable this by setting the
dremio.deltalake.enabled support key to
true. See Support Keys for instructions on setting support keys.
Cancelled queries may not be terminated correctly
Resolved by checking if the query was canceled when
RelMetadata is generated.
Dremio was reading beyond the end of an organization’s Parquet files, causing memory issues.
If Dremio reaches the end of a Parquet file and attempts to read any further, an exception is thrown to stop the service.
Users with the MODIFY privilege for a space would encounter permission errors when listing objects due to the privilege being treated as inheritable by child objects the user lacked access to. The MODIFY privilege now only functions at the space/source level and when listing objects is not treated as inheritable by any children.
The client connection was closed abruptly, canceling a query while outstanding messages for a coordinator node had not yet been acknowledged. This prevented any future query messages to the same coordinator from being acknowledged, and thus future queries could not progress beyond the queue.
All messages are acknowledged by the coordinator, even in the case of a query failing.
Companies upgrading from Dremio 4.X would encounter deserialization errors with expired reflections after upgrading to 16.X. Two new options have been added to prevent a query from refreshing an expired plan or reflection after upgrading. The following process should be followed if a reflection is set to expire and then a VDS is created with a raw reflection and query:
jars/3rdparty/calcite-core-1.16.0*.jarfile with the corresponding version for v16.0.
Now the reflection should refresh after starting.
After upgrading to 16.4, users encountered delays with regard to queries, planning time, and reflections. Multiple fixes have been implemented to improve query, planning, and reflection times.
A coordinator exhibited multiple queries had failed to be canceled, which helped to reveal a memory leak in one of Dremio’s services that occurs when canceling queries. Because of the canceled query, the completable future result is not consumed, which ultimately results in a memory leak.
Multiple improvements have been made to counteract this behavior, which includes:
Users able to run queries on temporary datasets cannot download results as the option is grayed out.
This issue has been resolved by adding permissions for temporary datasets.