On this page

    Source Metadata Policy

    This topic provides the configuration properties for source metadata policies.

    {
      "authTTLMs": Number,
      "datasetRefreshAfterMs": Number,
      "datasetExpireAfterMs": Number,
      "namesRefreshMs": Number,
      "datasetUpdateMode": String ["PREFETCH", "PREFETCH_QUERIED", "INLINE"]
    }
    

    Property Description

    NameTypeDescription
    authTTLMsNumberAuthorization expiry in milliseconds.
    datasetRefreshAfterMsNumberHow often (in milliseconds) to refresh the metadata of datasets in the source. The value must be >= (greater than or equal to) 3600000 (one hour or more).
    datasetExpireAfterMsNumberHow long (in milliseconds) to keep the metadata before it expires. The value must be >= (greater than or equal to) 3600000 (one hour or more).
    namesRefreshMsNumberHow often (in milliseconds) to discover new datasets in the source.
    datasetUpdateModeStringMust be either PREFETCH, PREFETCH_QUERIED, or INLINE.

    datasetUpdateMode options

    • PREFETCH - Dremio updates details for all datasets in a source. This mode increases query performance as less work needs to be done at query time.
    • PREFETCH_QUERIED - Dremio updates details for previously queried objects in a source. This mode increases query performance as less work needs to be done at query time for these datasets.
    • INLINE - Dremio updates details for a dataset at query time. This mode minimizes metadata queries on a source when not used, but might lead to longer planning times.