On this page

    22.0.0 Release Notes (June 2022)

    Known Issues

    • When multiple SQL statements are executed in the SQL Runner, any jobs that may have failed are not listed in the job summary below the SQL Editor.

    • The fields parameter is not returned for tables in external sources when fetching table details via /api/v3/catalog{id} if the table has not been queried.

    What’s New

    • Dremio now supports row-access and column-masking policies for fine-grained control over user query access to sensitive tables, views, and columns. This allows administrators to dynamically exclude or mask private data at the column and row levels prior to query execution and without physically altering the original values.

    • This release adds to existing Iceberg DML capabilities allows users to run DELETE, UPDATE, MERGE, and TRUNCATE statements against Iceberg tables. See SQL Commands for Apache Iceberg Tables for more information.

    • You can now add Azure Data Explorer (ADX) as a database source in Dremio. For more information, see Azure Data Explorer.

    • Autocomplete is now available in the SQL Editor. When enabled, autocomplete lets you view and insert possible completions in the editor using the mouse or the arrow keys with Tab or Enter. Autocomplete can provide suggestions for SQL keywords, catalog objects, and functions while you are constructing SQL statements. Suggestions depend on the current context. The autocomplete feature can be enabled or disabled for all users under Settings > SQL.

    • The SQL Runner now allows you to save your SQL as a script. See Querying Your Data for more information.

      • Script owners are indicated with a small orange flag next their username. Script owners cannot be removed or have their privileges changed.

      • You can share scripts with others in your organization by adding users and assigning privileges to View, Modify, Manage Grants, or Delete.

      • When adding or modifying script privileges, the View privilege is enabled automatically if any of the other privileges are enabled

      • The option to save a script will be disabled if the user already has 100 scripts, which is the maximum per user.

    • Added support for internal schema using SQL commands, which lets the user override the data type of a column instead of using the type that Dremio automatically detected.

    • Iceberg is the default CTAS format for all filesystem sources in Dremio 22.0.0+.

    • The DATEDIFF and ADD_MONTHS Hive functions are supported in queries.

    • You can now create SQL scalar functions that take input parameters and return a single values.

    • The option to enable Arrow caching in advanced reflection settings has been removed because Arrow caching is not supported with unlimited splits.

    • ALTER TABLE commands are now supported to add, drop, or modify columns in MongoDB sources.

    • Users can now resize the Data panel when viewing the SQL Editor on the Datasets or SQL Runner page.
    • The Zookeeper version used in Kubernetes deployments has been upgraded to 3.8.0 to address known security vulnerabilities. As part of any upgrade, it is best practice to back up configuration files and stateful volumes.

    • Fields in MongoDB tables can be converted to VARCHAR using internal schema (ALTER commands) and incompatible data types will fall back to VARCHAR instead of failing when querying MongoDB tables.

    • If the user tried to cancel a completed job, Dremio was returning an internal server error. The message now indicates that the job may have already completed and cannot be cancelled.
    • Improved logging and now providing a more meaningful error message when invalid characters are encountered in a password or PAT.
    • In this release, we have updated various elements of the Dremio UI to provide a more uniform and intuitive user experience.
    • In this release, many messages provided in Dremio have been updated to provide information that is more accurate and more helpful.
    • Iceberg now supports metadata functions for inspecting a table‚Äôs history, snapshots, and manifests.

    • New commands are available for ALTER keyword. By issuing the ALTER FOLDER or ALTER SPACE command, user can now set reflection refresh routing at folder and space level.

    • Users can now add a primary key to a table or drop a primary key with the following commands:
      alter table <table name> add primary key (col1, ...)
      alter table <table name> drop primary key

    • Dremio’s OAuth config now supports access_token as a valid token type to provide identity when authenticating via OpenID Connect SSO.

    • Dremio now supports OIDC + LDAP mode, which allows the use of OpenID Connect (OIDC) for authentication while still using LDAP for user and groups lookup.
    • In this release, json-smart was upgraded to version 2.4.8 to address CVE-2021-27568.
    • Updated the Postgres JDBC driver from version 42.2.18 to version 42.3.4 to address CVE-2022-21724.
    • In this release, the Apache Arrow version has been upgraded to 8.0.0 to address issues with some current functions and add support for new functions.
    • FasterXML/Jackson was upgraded to version 2.13.2 in Parquet to address a number of vulnerabilities.
    • This release includes a new consent page where you can permit Tableau to access resources on your behalf when connecting via Tableau SSO.

    • Along with ROW and ARRAY keywords, STRUCT and LIST keywords are now supported to represent complex data types:
      STRUCT < x : BIGINT, y : LIST < BIGINT >>, LIST <STRUCT < x : INT >>

    • This release adds support for the MODIFY privilege on SYSTEM that will allow non-admin users to manage Node Activity, Engines, Queues, Engine Routing, and Support Keys.

    • Dremio now supports MODIFY COLUMN on MongoDB sources, and the internal schema changes will not be erased by metadata refresh.

    • The SELECT privilege can be granted to users and roles on specific system tables, allowing those users to view the specified tables.

    • In the job details page, the automatic truncation message will appear in the job summary if a query’s output was truncated.

    • Dremio admins can allow or disable the creation of local users by adding the services.coordinator.security.permission.local-users.creation.enabled:<flag> setting to dremio.conf. Set the flag to true to allow local users or false to disable the creation of local users.

    • Added the UPLOAD privilege, letting non-admin users upload files to their home space. This privilege can be overridden if the ui.upload.allow support key is disabled.

    • Added a plus button to the upper-right corner of the page for spaces that allows users to quickly add a new folder, table, or view. For user home spaces, the button also allows you to upload files.
    • Dremio now supports SSO authentication from Tableau. See Tableau for more information about supported versions and configuration steps.
    • If the context is truncated in the SQL Editor, you can now hover the cursor over the field and the full context will be displayed in a tooltip.
    • User can now see the wiki for a folder if they can access the folder, even if implicitly via a shared dataset that is nested inside.

    • This release includes a new argument for the dremio-admin clean CLI command to purge dataset version entries that are not linked to existing jobs. See Clean Metadata for more information.

    • Users who have been granted the CREATE ROLE privilege can view and update role members.

    Issues Fixed

    • Running ALTER PDS to refresh metadata on a Hive source was resulting in the following error: PLAN ERROR: NullPointerException

    • Some queries were taking longer than expected because Dremio was reading a STRUCT column when only a single nested field needed to be read.

    • On first run, some queries were failing with an assertion error at the planning stage when a complex type was defined within a view.

    • Following the upgrade from Dremio 20.x to 21.0.0 and if Nessie was in use, metadata refreshes were failing with Unknown type ICEBERG_METADATA_POINTER.

    • The Tableau and PowerBI buttons were not showing up or remaining hidden as expected, and they are now enabled all the time in the SQL Editor.

    • After enabling Iceberg, files with : in the path or name were failing with a Relative path in absolute URI error.

    • Reflection refresh jobs were consuming too much peak memory on each executor node.

    • CAST operations were added to pushed down queries for RDBMS sources to ensure consistent data types, and specifically for numeric types where precision and scale were unknown. In some cases, however, adding CAST operations at lower levels of the query was disabling the use of indexes in WHERE clauses in some databases. Dremio now ensures that CAST operations are added as high up in the query as possible.

    • Following an upgrade, queries with TO_NUMBER(_Column_,'###') were failing.

    • In environments with high memory usage, if an expression contained a large number of splits, it could eventually lead to a heap outage/out of memory exception.

    • Fixed an issue that was causing the following error when trying to open a view in the Dataset page: Some virtual datasets are out of date and need to be manually updated.

    • When using Postgres as the data source, expressions written to perform subtraction between doubles and integers, or subtraction between floats and integers, would incorrectly perform an addition instead of the subtraction.

    • When running a specific query with a HashJoin, executor nodes were stopping unexpectedly with the following error: SYSTEM ERROR: ExecutionSetupException

    • At times, in Dremio’s AWS Edition, the preview engine was going offline and could not be recovered unless a reboot was performed.
    • Dremio was generating a NullPointer Exception when performing a metadata refresh on a Delta Lake source if there was no checkpoint file.
    • Partition expressions were not pushed down when there was a type mismatch in a comparison, resulting in slow queries compared to prior Dremio versions.
    • Fixed an issue that was causing large spikes in direct memory usage on coordinator nodes, which could result in a reboot.

    22.0.3 Release Notes (Enterprise Edition Only, July 2022)

    Enhancements

    • Amazon OpenSearch is now supported as a source on Dremio’s AWS Edition. For more information, see Amazon OpenSearch Service.
    • Azure Data Lake Storage (ADLS) Gen1 is now supported as a source on Dremio’s AWS Edition. For more information, see Azure Data Lake Storage Gen1.
    • Elasticsearch is now supported as a source on Dremio’s AWS Edition. For more information, see Elasticsearch.