2.1 Release Notes
Approximate count distinct acceleration
Dremio now supports accelerating count distinct queries based on an approximation based algorithm (HLL). This provides a faster and more memory efficient way of providing distinct counts. This is especially useful in high cardinality scenarios with very large datasets. Internally, this capability uses Dremio’s
Ability to define “Selected Measures” for aggregation reflections
Dremio now supports the ability to set/unset measures per field when setting up aggregation reflections. Previously, Dremio would include
MAX as set measures by default for any field. You can now set/unset these (including
Approximate count distinct) separately per field with the defaults being set for the COUNT and SUM measures.
External reflection support for RDBMS data sources
External reflections are now supported for all Dremio sources, including RDBMSs.
Datasets can now be configured to have associated reflections never expire
Previously, you needed to set a really high expiration value to avoid reflections expiring. A “Never Expire” option was added to make this more explicit.
Web Application and APIs
Space creation is now admin only by default
Added a new support setting
ui.space.allow-manage that overrides the default, allowing non-admin users to manage spaces.
REST API-based security management
Improved the Dremio Catalog API to allow managing security on all endpoints.
Improved source form UI experience
Improved workflow in the UI to streamline defining new data sources. Added logically separated sections that are intended to make the experience of adding new data sources easy.
Allow resetting of advanced options in the admin UI
Added ability to reset advanced options to their defaults in the UI.
Allow disabling of file uploads
Added support to disable the file upload functionality. This allows admins to monitor space usage and enforce quotas around it.
Job Profiles now show physical bytes read Job Profiles now show physical bytes read instead of Arrow in-memory representation size.
Cmd+click and middle button click on New Query opens query editor in a new browser tab This fixed an issue where “cmd+click” is supposed to open a new query screen in a new window, but was incorrectly opening it up in the same window.
Coordination and Metadata
Query Planner Changes
Depending on the query, performance characteristics may be impacted specifically in two main ways:
Correlated queries (subquery comparison to a top query): Dremio may now do fewer JOINS and scans of tables so it should improve performance.
Join optimizations: Dremio now groups multiple joins together and orders them in the most efficient way allowing us to capture more use cases. Please note the order of joins in your plan may be different and in many cases this will improve performance, but could result less-than-optimal performance for other edge cases.
New Dremio YARN Autobundler: The YARN bundle is now auto-generated at runtime.
- The installation tarball is now half the size of what it used to be and therefore it should be quicker to download.
- Any file added to Dremio’s Classpath will automatically be copied to YARN now.
Allow experimental use of transitive join predicate optimization
Filtering on join key now pushes down on both sides of the join to minimize the records read and then try to join. This can be enabled by the following support key
Support for Elasticsearch version 6, which is currently the latest version.
Hive ORC Performance
Improved performance and memory characteristics when working with ORC files that reside in Hive sources. Dremio now also better handles predicate pushdowns.
Support for SSL encryption when connecting to Oracle
Added support for connecting to Oracle over TLS using the JDBC thin driver. This is certified to work with Oracle Database 12c Release 1 (12.1) and (11.x.0.x). However, it is not certified to work with older, unsupported database releases, such as 10.2.x, 10.1.x, 9.2.x, and 9.0.1.x.
Added support for S3 buckets in AWS GovCloud Added support for S3 buckets in AWS GovCloud as a source and as distributed storage for reflections.
Compression support for Elasticsearch traffic Added support for compressing Elasticsearch responses to minimize network traffic.
Improved stats estimation for Hive tables
Due to frequent problems with bad/outdated table stats on Hive tables we changed the default to rely on Dremio’s own table statistics estimates and not Hive’s.
Improved small file performance on S3/ADLS
Enhanced performance for small files on high latency stores like S3 & ADLS.
Simplified memory configuration via
DREMIO_MAX_MEMORY_SIZE_MB in dremio-env
Setting this configuration will allow Dremio to automatically determine the best allocation between HEAP a DIRECT memory depending on the node type.
Coordination and Metadata
Cannot re-add a previously deleted user
A previously created user that is subsequently deleted can now be re-created.
Join optimization not working as expected in certain scenarios Join optimization has been improved in scenarios where projects or filters exist between joins.
Filters specified as a part of
<t> JOIN <t> ON [...] clause would not be evaluated before join operation
Fixed scenario where a filter condition on an inner join was not pushed below the join.
Hive tables with large amount of partitions would take up too much space in Dremio’s metastore
Improved meta-data handling for Hive tables with large number of partitions.
ODBC slowness with Tableau in some situations
Improved performance of schema retrieval for Tableau users.
Some changes to source configuration causes datasets contained to loose formatting
Resolved an issue where an invalid source config causes existing datasets to loose formatting.
Incorrect join output record estimation causes query to be over-parallelized
Resolved issues with join planning that causes queries to be run for longer with excessive resource requirements.
Date/time functions are not pushed down as expected
Improved handling of
CURRENT_TIME to ensure optimal pushdowns of date/time functions.
Reflection job completion is not visible in reflection history
After a reflection refresh finishes successfully it now shows up in the reflection’s job history with the proper reflection id.
UI incorrectly shows non-bigint fields as options for incremental update columns
Currently the UI suggests incremental refresh on date and timestamp for Mongo and Elastic sources, but fails to refresh associated reflections with error.
Creating a reflection on a table with decimal fields will fail
Dremio no longer fails expanding the materialization when creating a reflection on a table with decimal fields
Reflection matching would occasionally fail with a silent StackOverflowError
Fixed an issue where a StackOverflowError would get generated when executing a query using the RANK function, if a matching reflection is present.
Acceleration page might appear hung without an indication of work-in-progress
Fixed issue where the reflections screen has no progress icon while loading reflections.
Reflections are incorrectly marked as invalid despite successful creation
Some reflections are marked as invalid due to NPE in
JoinAnalyzer code path, despite being successful. This issue has ben fixed.
Fixed issue where reflection matching sometimes doesn’t match valid joins
Improved JOIN reflection substitution matching to consider a wider set of alternatives.
Failures when matching a reflections affects other reflections for that query
Resolved an issue where a failure happening in post-substitution processing resulted in all substitutions failing to be registered with the planner. Now, if the failure is only happening during the processing of a single candidate only that candidate is excluded.
Some queries using ANSI JOIN syntax may not be accelerated
Resolved an issue where a query against a table with an INNER JOIN and no corresponding WHERE clause is NOT accelerated.
Web Application and APIs
UI incorrectly showed wrong port when creating HDFS sources
Changed the default port for HDFS showing up in the UI from 9000 to 8020, which is the default port for HDFS namenode.
Changing refresh policy shows incorrect warning as part of source creation Changing refresh policy no longer shows warning as part of source creation.
No error shown when failing to edit a source
Users are now presented with correct error message.
Data lineage graph is formatted incorrectly
Fixed an issue where the lineage graph wasn’t being rendered correctly.
Cannot download query if using a relative path (via query context)
Fixed issue with using relative paths when downloading a CSV/JSON file gives “Path not found”.
If the PDS a VDS is based on is no longer available, user cannot edit original sql
Fixed issue where if the PDS a VDS is based on is no longer available the user couldn’t edit the original sql
UI slowness when displaying counts for sources with 100K+ objects
UI behavior has been updated to show up to 500 datasets. If number of datasets in a source is 500 or more then the UI will display “500+” or “-”.
Slow list rendering in the UI with sources that have many objects Removed the descendant and dataset count columns in dataset listings to improve UI performance. Both are still accessible per dataset by clicking on the dataset icon to the left of each dataset.
Massive dataset listings in the UI could use too much HEAP memory Performance improvement for massive dataset listing for sources and space
Cannot “Edit Original SQL” when there are errors when querying VDS in UI
The “Edit Original SQL” button should now be visible when VDS returns with error when a PDS doesn’t have certain fields available any more.
Renaming a VDS would sometimes cause error when trying to edit definition
This fixed an issue where altering a VDS definition and then renaming the VDS name breaks the VDS and errors out.
UI doesn’t show error when it cannot reach Dremio coordinators
UI now correctly shows a message when it cannot connect to Dremio coordinators.
Dremio incorrectly ignores cast to varchar on datetime fields against SQL Server
Dremio will now keep cast to varchar intact when pushing down to SQL Server.
SSL based Elasticsearch sources go unusable if Dremio crashes Fix issue with Dremio connecting to Elasticsearch when self-sign certificates are used in SSL connection.
Fixed timestamp/date/time comparison in Elasticsearch Painless scripts Fix literal comparison for timestamp/date/time types.
Redshift source connections occasionally become stale
Fixed a scenario where we occasionally received a stale connection when using Redshift causing dataset refreshes to fail and return no tables.
Truncate function is incorrectly translated when working with Oracle sources
Translation of the TRUNCATE function in Oracle is now fixed.
Sub-optimal memory characteristics when retrieving aggregation results from Elasticsearch Elasticsearch reader has been optimized to leverage off-heap memory when reading aggregation results
SQL Server date/time field pushdowns lose precision
There was an issue with SQL Server pushdown losing milliseconds precision. These are now handled correctly.
DAYOFWEEK(…) and its variants incorrectly return Monday as 1, etc.
Currently DAYOFWEEK in Dremio behaves as follows: 1=Monday,…,7=Sunday. Changed it so that 1=Sunday and 7=Saturday.
Start index of 3-arg POSITION() function is treated as zero-based
Fixed start index argument of POSITION function to be 1-based.
Query failure when columns of a dataset being used downstream are changed
Fixed an exception caused when a VDS dependent on another VDS and the columns of the underlying VDS are changed.
Schema learning would not function as expected on Parquet datasets Fixed an issue where the schema learning wasn’t working as expected on some Parquet datasets.
Failure while reading sort spilling files error when doing sort spilling
Fixed memory leak when reading sort spilling files fails.
Issue with reading parquet page blocks of size > 1024
Dremio can now read parquet page blocks of size > 1024 such as the ones generated by
Filter pushdown issues when using Impala written Parquet files
Dremio now specifically accommodates varbinary & varchar fields written by Impala during filter pushdowns.
#2.1.4 Release Notes
SQL Server sources cannot be removed under some scenarios and would result in Dremio hanging
Dremio now allows SQL server to be removed successfully; even after a restart.
If the user had any unsaved changes “Edit Original SQL” would sometimes not work as expected
‘Edit Original SQL’ should now go into edit mode for the latest version of the dataset.
Issue on Hive 1.2.1 with SQL Standard-based Authorization enabled where trying to fetch the functions ‘get_functions’ would fail
Dremio now avoids fetching the get_function function via modification to the Hive code.
A JOIN query with FILTER that gets pushed down to Oracle data sources can take a very long time to execute
Joins with filter now complete in a more reasonable amount of time.
Dremio cluster can become unstable while upgrading to 2.0.11 or 2.1.2
Improved stability by resolving an issue with the size of the KV store growing unnecessarily.
Accessing Hive transaction tables in ORC format that are bucketed and are type transactional creates unreadable files in HDFS
Dremio can now read/query hive transactional table with bucketing enabled.
Dremio community edition for Windows is unable to execute queries
Fixed an issue with the communication between coordinator and executor nodes where the query planning would complete, but execution would never begin.
Creating external reflection would give “unexpected error occurred” message
Dremio now allows external reflections to be successfully created.
Agg reflections can fail with error “Failure while reading sort spilling files, on mapr cluster”
Fixed Agg Reflections failure due to sort spill file I/O issue on MapR.
#2.1.6 Release Notes
RAW reflections in version 2.1.4 were not being matched properly
Raw reflections are now successfully matched and used for acceleration.
Agg reflections in version 2.1.4 were not being substituted as expected
Agg reflections are now successfully substituted as part of the query plan.
Dragging the mouse in the Histogram UI screen doesn’t behave properly
Moving the mouse on the Histogram screen now allows the chart to be moved as expected.
If a VDS name contains ‘/’, the upgrade from 2.0.5 to 2.1.4 will fail.
Fix upgrade issue caused by VDS name containing ‘/’
Compatibility with CentOS 6 in Dremio v2.1 is broken
Fixed CentOS 6 compatibility issue with a jna library built with GLIBC2.14.
Reindexing on startup after a crash was slow performing
Improved the performance of reindexing on startup after a crash.
Rpc Exception system errors were being caused by out-of-memory
Resolved an issue due to a memory leak.