4.6 Release Notes
What's New
AWS Edition
Dremio offers a new hourly paid listing in AWS Marketplace.
Dremio displays a checkmark in a new
Enterprise
column on theProjects
page for projects that were last run with Dremio Enterprise features enabled.
Other Enhancements
Dremio users can launch Microsoft Power BI by clicking an icon on the Client toolbar on the Dremio
Datasets
page. The Client toolbar allows users to launch Tableau and Power BI from Dremio. Dremio administrators can configure which clients appear on the toolbar for a Dremio project in theClient Tools
pane of theSupport Settings
page and with a new REST API endpoint.Dremio supports MongoDB 4.2.
The Power BI connector is now GA.
Dremio supports AWS Glue Catalog as a data source and automatically synchronizes all databases and tables in the Glue Data Catalog that reside in AWS S3. Dremio supports the following data formats for AWS Glue:
- Parquet
- ORC
- Delimited text files (CSV/TSV)
A new user interface for the
Elastic Engines
page:- On the
Admin
page underWorkload Management
,Provisioning
is nowEngines
. - The
Engines
page displays a list of all available engines for the project. Dremio administrators can add and remove workers from an engine, as well as edit and delete engines. - The
Engine Details
page contains aQueues
tab, which displays the queue memory limit and the job memory limit per node.
- On the
A new
CONVERT_TIMEZONE
SQL function that converts a timestamp to a specified timezone. Two new system tables,sys.timezone_names
andsys.timezone_abbrevs
, list Dremio-supported timezone names and abbreviations. The function accepts three parameters:sourceTimezone
(optional)destinationTimezone
timestamp
If the
sourceTimezone
parameter is not present, Dremio assumes the timestamp provided in the third parameter is in UTC format.The
sourceTimezone
anddestinationTimezone
parameters accept any of the following values:- timezone name from the
sys.timezone_names
- timezone abbreviation from
sys.timezone_abbrevs
- offset, such as
+02:00
Upgraded to the latest version of Apache Arrow, 0.17.1.
Sorting now utilizes lz4 compression for improved performance.
A new environment variable in
dremio-env
,DREMIO_GC_OPTS
, that specifies which Java garbage collector Dremio uses. Dremio supports the Garbage First Garbage Collector (XX:+UseG1GC
) and the Parallel Collector (-XX:+UseParallelGC
). By default, Dremio uses G1 GC.Dremio detects if a conflict exists between the garbage collection configuration specified with the new
DREMIO_GC_OPTS
environment variable and theSet Up YARN
dialog. If garbage collection is configured for YARN, Dremio uses that value in the event of a conflict. If the garbage collection is not configured for YARN, Dremio uses the value specified byDREMIO_GC_OPTS
.Improved error messages for the JDBC Connector.
Changed Behavior
Users without permissions on a data source can no longer create a table from query results (CTAS). Dremio now checks for user permissions on a data source for the following queries:
createTableAs
createEmptyTable
createNewTable
createView
updateView
dropView
dropTable
truncateTable
addColumns
dropColumn
changeColumn
The Qlik Sense client application again appears on the
Analyze With
menu on theDatasets
page, but no longer appears on the Client toolbar.The
TO_TIMESTAMP
SQL function supports the timezone abbreviations in the newsys.timezone_abbrevs
system table.Data Reflections configured for incremental refreshes only perform a full refresh after changes to the schema that impact the Data Reflection. Changes to the schema that do not impact the Data Reflection no longer inititate a full refresh.
The Power BI Connector prioritizes
Dremio.pqx
rather thanDremio.mez
when both files are installed. Use of.mez
files is deprecated.
Fixed Issues in 4.6.0
NOT IN
clause sometimes returns NULL records
Fixed an issue that caused the NOT IN
clause to return NULL records when the clause contains five or more values.
Sorting performance may be impacted by compression and small batch sizes
Fixed by disabling an unncessary flush.
Dremio fails to restart after upgrading from Dremio 4.1.8 to 4.5 on AWS Edition
Fixed by removing previously configured elastic engines in Dremio during the upgrade.
JDBC storage plugin fails while setting up SQL queries
Fixed a bug that occurred when aliases are used in the ORDER BY
SQL clause.
Unable to load credentials on Amazon EKS deployments
Fixed by retrying when AWS SDK returns an SdkClientException
.
Unable to delete a Dremio user with special characters in the username
Fixed an issue with escape characters.
Unable to restore a standalone Dremio deployment using AWS S3 for distributed storage to another standalone deployment also configured for distributed storage with S3
Fixed misleading error message during restore operation.
Tableau Server on Linux cannot connect to a Dremio deployment when SSL/TLS is enabled
Fixed by upgrading the SSL driver to version 1.5.0.1001 in the Dremio ODBC driver for Linux.
JDBC storage plugin sometimes fails while setting up queries on Postgres
Fixed issue that added superfluous collations.
Power BI Connector generates an error when using text filters with non-English characters
Fixed by overriding SQLGetTypeInfo
to return only one type for SQL_WVARCHAR
and SQL_WCHAR
.
UnsupportedOperationException
while trying to read HDFS datasources in Parquet format
Fixed issue with case sensitivity in column names of HDFS datasources in Parquet format.
4.6.1 Release Notes
Enhancements
New advanced option for AWS S3 data sources,
Enable file status check
, and new property for metadata storage indremio.conf
:debug: { dist.s3_file_status_check.enabled: enabled }
These options control whether Dremio verifies that a file exists in AWS S3 and the distributed storage for a data source, respectively. Both are enabled by default. If users notice failed
LOAD MATERIALIZATION
orDROP TABLE
data acceleration jobs when using AWS S3 for distributed storage, disabledist.s3_file_status_check.enabled
indremio.conf
and disable theEnable file status check
advanced option on the data source.New metric,
NUM_COLUMNS_TRIMMED
, reports the number of trimmed columns in Parquet-formatted files.
Fixed Issues in 4.6.1
Validation error with java.io.FileNotFoundException
when refreshing a Data Reflection
Fixed by disabling the new Enable file status check
advanced option for AWS S3 data source and disabling the debug.dist.s3_file_status_check.enabled
property in dremio.conf
.
Queries on Dremio metadata containing a WHERE clause with both LIKE
and OR
operators return incorrect results
Fixed by correctly pushing down OR
query filter.
Executor nodes fail with ForemanException
Fixed by removing unnecessary columns and rowgroups from footers of Parquet files.
When asynchronous access is disabled, Dremio is unable to gather medata from the footers of Parquet files
Fixed by reverting to a known working parquet footer.
Dremio crashes with java.io.FileNotFoundException
Fixed issue with data consistency during refreshes of Data Reflections for AWS S3 data sources.
Inconsistent job status reported in job profile and job details
Fixed by asynchronously handling completion events from executor nodes.
Superfluous columns are not trimmed while scanning Data Reflections
Fixed by adding a handler method.
4.6.2 Release Notes
Fixed Issues in 4.6.2
Queries fail with CONNECTION ERROR: Error setting up remote fragment execution
Fixed issue where parallelizer picked executor nodes outside of the selected engine.