Dremio Cloud
Dremio Software
 
On this page

    Source Metadata Policy

    This topic provides the configuration properties for source metadata policies.

    Source metadata policy 
    {
  "authTTLMs": Number,
  "datasetRefreshAfterMs": Number,
  "datasetExpireAfterMs": Number,
  "namesRefreshMs": Number,
  "datasetUpdateMode": String ["PREFETCH", "PREFETCH_QUERIED", "INLINE"]
}

    Property Description

    Name Type Description
    authTTLMs Number Authorization expiry in milliseconds.
    datasetRefreshAfterMs Number How often (in milliseconds) to refresh the metadata of datasets in the source. The value must be >= (greater than or equal to) 3600000 (one hour or more).
    datasetExpireAfterMs Number How long (in milliseconds) to keep the metadata before it expires. The value must be >= (greater than or equal to) 3600000 (one hour or more).
    namesRefreshMs Number How often (in milliseconds) to discover new datasets in the source.
    datasetUpdateMode String Must be either PREFETCH, PREFETCH_QUERIED, or INLINE.

    datasetUpdateMode options

    • PREFETCH - Dremio updates details for all datasets in a source. This mode increases query performance as less work needs to be done at query time.
    • PREFETCH_QUERIED - Dremio updates details for previously queried objects in a source. This mode increases query performance as less work needs to be done at query time for these datasets.
    • INLINE - Dremio updates details for a dataset at query time. This mode minimizes metadata queries on a source when not used, but might lead to longer planning times.