4.9 Release Notes

What’s New

Runtime Filtering - Non-partitioned columns

Expanded Runtime filtering to support predicates on non-partitioned columns.

Other Enhancements

  • Dremio provides a heap monitor for coordinator nodes to protect from heap outages. The heap monitor cancels queries that are using the maximum heap size. The feature is enabled by default.

  • Improved query planning to significantly reduce logical planning times for queries with a large number of data reflections, multi-level joins, or aggregations.

  • Improved the performance of queries when caching reflections.

  • AWS Edition supports smaller EC2 instance types for Engines making it easier to use Dremio for trial and evalutation purposes.

Fixed Issues in 4.9.0

UI does not allow to view all spaces when moving a VDS
Fixed the issue by displaying the scrollbar.

Users cannot select the dataset after previewing the schema in global search results
Fixed the global search results UI.

Some reflections were marked cannot accelerate in the UI and does not work
Resolved by returning the correct type for count function during projectable aggregate conversion.

Dremio throws a ClassCastException when a date type is used in the where clause of a query
Fixed by internally catching the ClassCastException.

UI displays a Something went wrong error when clicking on the graph of a VDS
Fixed by correcting the overlapped VDS condition in the UI.

Dremio returns the 0001-01-01 as 0000-12-13 date field on ORC tables
Fixed by not pushing down filters for date columns.

Incremental refresh on aggregate reflection does not aggregate
Fixed by adding the rollup aggregate during incremental refresh.

User encountered “This query cannot be planned - possible due to use of an unsupported feature” while executing a query
Fixed the error message, now reads: “Query contains too many correlated subqueries”.

User encountered reflection materialization in RUNNING state that never completes
Fixed deleted reflection on coordinator restart.

Job API returns fewer results when using a particular offset
Fixed by reading multiple arrow files based on offset and limit values.

Metadata refresh on source stops refreshing the rest of the datasets when one dataset’s number of files exceeds dremio.store.dfx.max_file limit
Fixed by issuing a warning message when a dataset exceeds dremio.store.dfx.max_file limit and continue refresh the remaining datasets.

User encounters ArrayIndexOutOfBoundsException when selecting a particular column in a query
Fixed the Dremio query planner issue.

Dremio returns a wrong record count when reflection is used in some queries
Fixed the issue by reworking materialization partition names.

Job status shows query still running while query is completed
Resolved the issue by correctly terminating job status.

dremio-admin restore fails with NPE when restoring from a faulty backup
Fixed an issue that prevents restoration of job_info files with empty dataSetPaths.

User encountered UnsupportedOperationException when querying parquet files stored on ADLS
Fixed the issue by not using an union type as a partition column.

User encountered sql2rel.RelDecorrelator.decorrelateRel when running a query
Fixed the issue by removing multiple constant groups by keys in aggregate before decorrelation during query planning.

User encountered query timeout due to memory issue
Resolved the issue by freeing up jmodel and ClassGenerator objects after code generation.

User encountered Cannot convert RexNode to equivalent Dremio expression during VDS creation
Fixed the issue by changing order of query planning for certain cases.

Inaccurate reported pendingTime or metadataRetrivalTime in queries.json
Fixed the issue by correcting the reported time.

Dremio engine stopped but executors still show as running
Fixed the issue by fixing the node listener logic.

Reflection creation succeed but reflection marked cannot accelerate in UI and does not work
Fixed the issue by using the correct type for count conversion.

User experienced slowness in Dremio performance
Resolved by releasing CodeGenerator object early to reduce heap usage

User encountered ClassCastException during query execution
Resolved the issue by catching the exception to prevent query from failing.

Long running JDBC queries incorrectly marked as Failed instead of Cancelled if user cancels them
Fixed the issue by correcting the query status.

Failed to access dataset in source with FileNotFoundException
Updated the error message to be invalid metadata (For example, the file was deleted.)

Fixed by relaxing some unnecessary validation checks.

Job status for a failed query mismatched between left and right panel on the job page UI in scale-out-coordinator setup
Fixed by properly handling the job status update.

Failed to access homespace due to change of the username which is case sensitive
Workaround is to delete the user’s homespace using the CLI tool: ./dremio-admin delete-user-homespace -u

Gandiva cache is not saving all project expressions
Fixed by caching all expressions (direct or eval).

UI failed to show all spaces when there is a long list of spaces or try to move a vds from one space to another
Fixed the UI scroll height to view all spaces.

Dremio helm chart v2 failed due to wrong type for executor.extraInitContainers
Fixed the context for extraInitContainers when templating the user provided value.

CVE-2020-13692: Vulnerabilities reported against PostgreSQL JDBC driver, version version 42.2.5
Resolved by updating the reference to the PostgreSQL JDBC driver to version 42.2.17.

4.9.1 Release Notes

What’s New

Preview: Arrow Flight

Dremio 4.9.1 offers a new Arrow Flight endpoint for Arrow Flight connections. The feature is enabled by default on port 32010. Arrow Flight enables high speed data transfer compared to ODBC/JDBC connections by utilizing the Apache Arrow format to avoid serializing and deserializing data.

Dremio provides sample Flight client applications at Dremio Hub. See the Arrow Flight documentation for more information about Arrow Flight.

Known Issues in 4.9.1

The Arrow Flight endpoint returns a buffer_index out of range error for union-typed columns.

Fixed Issues in 4.9.1

Fails to connect to PostgreSQL servers with GSSAPI authentication enabled.
Resolved by updating the reference to the PostgreSQL JDBC driver to version 42.2.18.

Wait times of 2 to 3 seconds for HDFS threads
Fixed by extracting file size and mtime while retrieving Hive Parquet splits.

4.9.3 Release Notes (Enterprise Edition Only)

Dremio recommends that users install Dremio 4.9.3 when upgrading to 4.9.x to ensure that new permissions are set correctly. The 4.9.0 and 4.9.1 releases of Dremio Enterprise Edition are no longer available.

After upgrading to Dremio 4.9.3+ from 4.9.0 or 4.9.1, Dremio recommends that administrators reset permissions for datasets, folders, spaces, and sources that were set using the user interface in 4.9.0 or 4.9.1.

Fixed Issues in 4.9.3

When Dremio users select the Can Edit permission in the user interface, Dremio doesn’t automatically select the Can Query permission, as well
Resolved by automatically selecting and locking the Can Query permission when a Dremio user selects the Can Edit permission.