Source Metadata Policy

This topic provides the configuration properties for source metadata policies.

{
  "authTTLMs": Number,
  "datasetRefreshAfterMs": Number,
  "datasetExpireAfterMs": Number,
  "namesRefreshMs": Number,
  "datasetUpdateMode": String ["PREFETCH", "PREFETCH_QUERIED", "INLINE"]
}

Property Description

Name Type Description
authTTLMs Number Authorization expiry in milliseconds.
datasetRefreshAfterMs Number How often (in milliseconds) to refresh the metadata of datasets in the source. The value must be >= (greater than or equal to) 3600000 (one hour or more).
datasetExpireAfterMs Number How long (in milliseconds) to keep the metadata before it expires. The value must be >= (greater than or equal to) 3600000 (one hour or more).
namesRefreshMs Number How often (in milliseconds) to discover new datasets in the source.
datasetUpdateMode String Must be either PREFETCH, PREFETCH_QUERIED, or INLINE.

datasetUpdateMode options

  • PREFETCH - Dremio updates details for all datasets in a source. This mode increases query performance as less work needs to be done at query time.
  • PREFETCH_QUERIED - Dremio updates details for previously queried objects in a source. This mode increases query performance as less work needs to be done at query time for these datasets.
  • INLINE - Dremio updates details for a dataset at query time. This mode minimizes metadata queries on a source when not used, but might lead to longer planning times.

results matching ""

    No results matching ""