4.6.1 Release Notes

Enhancements

• New advanced option for AWS S3 data sources, Enable file status check, and new property for metadata storage in dremio.conf:

debug: { 
  dist.s3_file_status_check.enabled: enabled
}

These options control whether Dremio verifies that a file exists in AWS S3 and the distributed storage for a data source, respectively. Both are enabled by default. If users notice failed LOAD MATERIALIZATION or DROP TABLE data acceleration jobs when using AWS S3 for distributed storage, disable dist.s3_file_status_check.enabled in dremio.conf and disable the Enable file status check advanced option on the data source.

• New metric, NUM_COLUMNS_TRIMMED, reports the number of trimmed columns in Parquet-formatted files.

Fixed Issues in 4.6.1

Validation error with java.io.FileNotFoundException when refreshing a Data Reflection
Fixed by disabling the new Enable file status check advanced option for AWS S3 data source and disabling the debug.dist.s3_file_status_check.enabled property in dremio.conf.

Queries on Dremio metadata containing a WHERE clause with both LIKE and OR operators return incorrect results
Fixed by correctly pushing down OR query filter.

Executor nodes fail with ForemanException
Fixed by removing unnecessary columns and rowgroups from footers of Parquet files.

When asynchronous access is disabled, Dremio is unable to gather medata from the footers of Parquet files
Fixed by reverting to a known working parquet footer.

Dremio crashes with java.io.FileNotFoundException
Fixed issue with data consistency during refreshes of Data Reflections for AWS S3 data sources.

Inconsistent job status reported in job profile and job details
Fixed by asynchronously handling completion events from executor nodes.

Superfluous columns are not trimmed while scanning Data Reflections
Fixed by adding a handler method.