4.6.1 Release Notes
Enhancements
- New advanced option for AWS S3 data sources,
Enable file status check
, and new property for metadata storage indremio.conf
:
debug: {
dist.s3_file_status_check.enabled: enabled
}
These options control whether Dremio verifies that a file exists in AWS S3 and the distributed storage for a data source, respectively. Both are enabled by default. If users notice failed LOAD MATERIALIZATION
or DROP TABLE
data acceleration jobs when using AWS S3 for distributed storage, disable dist.s3_file_status_check.enabled
in dremio.conf
and disable the Enable file status check
advanced option on the data source.
- New metric,
NUM_COLUMNS_TRIMMED
, reports the number of trimmed columns in Parquet-formatted files.
Fixed Issues in 4.6.1
Validation error with java.io.FileNotFoundException
when refreshing a Data Reflection
Fixed by disabling the new Enable file status check
advanced option for AWS S3 data source and disabling the debug.dist.s3_file_status_check.enabled
property in dremio.conf
.
Queries on Dremio metadata containing a WHERE clause with both LIKE
and OR
operators return incorrect results
Fixed by correctly pushing down OR
query filter.
Executor nodes fail with ForemanException
Fixed by removing unnecessary columns and rowgroups from footers of Parquet files.
When asynchronous access is disabled, Dremio is unable to gather medata from the footers of Parquet files
Fixed by reverting to a known working parquet footer.
Dremio crashes with java.io.FileNotFoundException
Fixed issue with data consistency during refreshes of Data Reflections for AWS S3 data sources.
Inconsistent job status reported in job profile and job details
Fixed by asynchronously handling completion events from executor nodes.
Superfluous columns are not trimmed while scanning Data Reflections
Fixed by adding a handler method.