Dremio Community

Configuring Cloud Cache

This topic describes how to configure cloud caching.

Support

Dremio supports cloud caching for Parquet files on the following data sources:

  • Amazon S3
  • ADLS (Gen 1)
  • Azure Storage (ADLS Gen 2) - v2 only
  • HDFS
  • Hive on S3, ADLS, Azure Storage, and HDFS

Tip
Dremio AWS Edition provides cloud caching without manual configuration.

Enable Caching

Enable cloud caching for supported data sources either when adding a new data source to your deployment or later by editing the data source.

To enable cloud caching:

  1. On the Datasets page, select a data set based on a supported data source.
  2. Click the gear icon, , to the right of the search bar.
  3. Click Advanced Options.
  4. In the Advanced Options pane, check Enable asynchronous access when possible.
  5. In the Cache Options pane, check Enable local caching when possible.
  6. Click Save.

The Max percent of total available cache space to use when possible option specifies the the disk quota that can be used by a data source on any single executor node. The default is 100% of the total disk space available on the mount point provided for caching.

The following screenshot displays the Edit Source dialog for an Amazon S3 data source:

Setup Cache Path and Directory

For the Dremio cluster, the following must be specified:

  • Database path - This is the path for the database directory to use for storing cached data.
  • Cache directories – This is the mount point (base directory) for storing all the data related to caching on the node. Cache is lost if the directory location is changed and the executor is restarted.

Configuring via dremio.conf

To provision cloud cache you can configure the following dremio.conf settings:

  • Local directory path – By default, the cache manager uses the local directory path if database or file system paths are not specified..
  • Database path – The executor.cache.path.db setting provides the database directory path. If you do not specify a database path, the local directory path is used.
  • File system path – The executor.cache.path.fs setting provides the file system cache directory. Note that for good performance, SSD/NVMe disks are recommended for cloud cache. If you do not specify a file system path, the local directory path is used.

Example dremio.conf

In the following dremio.conf example, a database path and four (4) file system paths are specified. Both the database and file system paths are optional. If these paths were not specified, the cache manager uses the local path (/mnt/resource/dremio/data). Dremio uses 70 percent of the total available disk space for the specified database and file system mount paths.

paths: {
  # the local path for dremio to store data.
  local: "/mnt/resource/dremio/data"

  # the distributed path Dremio data including job results, downloads, uploads, etc
  #dist: "pdfs://"${paths.local}"/pdfs"
}

services: {
  coordinator.enabled: false,
  coordinator.master.enabled: false,
  executor.enabled: true
  executor.cache.path.db : "/mnt/cachemanagerdisk/db",
  executor.cache.path.fs : [ "/mnt/cachemanagerdisk/dir1","/mnt/cachemanagerdisk/dir2","/mnt/cachemanagerdisk/dir3","/mnt/cachemanagerdisk/dir4"]
}
zookeeper: "lak-azure-perf:"${services.coordinator.master.embedded-zookeeper.port}

Setup Caching for Reflection Data

If reflection data is in cloud store and if want to use caching for reflection data:

  1. Set dist.caching.enabled: true in dremio.conf under the debug section.
  2. Restart the cluster. This step is required for the change to take effect.
paths: {
  # the local path for dremio to store data.
  local: ${DREMIO_HOME}"/data"

  # the distributed path Dremio data including job results, downloads, uploads, etc
  dist: "dremioS3:///qa1.dremio.com/testdata/c3_dist/"
}

services: {
  coordinator.enabled: false,
  coordinator.master.enabled: false,
  executor.enabled: true
}
debug: {
  # Enable caching for distributed storage, it is turned off by default
  dist.caching.enabled: true,
  # Max percent of total available cache space to use when possible for distributed storage
  dist.max.cache.space.percent: 100
}

Best Practices

The following describes best practices for cloud caching:

  • Use SSD/NVMe disks for best performance.
  • Provide disks with sufficient space to benefit from caching. Note that the cache will evict unused data when required.
  • To add capacity to the cache, add additional disks to the executor.cache.path.fs property in the dremio.conf file. Note that removing a disk is not supported.
  • To improve performance while using reflections, enable caching on your distributed store. By enabling caching on your distributed store, caching of reflection data is also enabled because Dremio stores data related to reflections, job results, download, and uploads on the distributed store.
  • When removing local caching on executor nodes, you need to:
    1. Remove the Cloud Cache options on the data source.
    2. Delete the Cache Manager database and file system folders.