Skip to main content
Version: 24.1.x

Configuring Memory

Memory is configured via the /etc/dremio/dremio-env file. When this configuration file is changed from the default, it must be updated on all nodes of interest.

note

In addition to memory, this configuration file is used to set Java options and log directories.

Maximum memory size allows Dremio to automatically determine the best allocation between HEAP and DIRECT memory depending on the node type. To change the default maximum memory, modify the following line:

DREMIO_MAX_MEMORY_SIZE_MB=16384

Alternatively, you can set HEAP and DIRECT memory instead of maximum memory. The maximum HEAP and DIRECT memory can be set separately.

  • If both are specified, then the maximum memory option is ignored.
  • If only one is configured, Dremio automatically determines the other, based on leftover memory.

Heap memory​

Heap memory is used for running Dremio server. To change the default maximum heap memory, modify the following line:

DREMIO_MAX_HEAP_MEMORY_SIZE_MB=8192

Direct memory​

Direct memory is used for query execution. To change the default maximum direct memory, modify the following line:

DREMIO_MAX_DIRECT_MEMORY_SIZE_MB=16384

If you see queries failing on an instance that's running out of memory, increasing the amount of memory that Dremio is able to consume may solve your problem.

caution

For the DREMIO_MAX_DIRECT_MEMORY_SIZE_MB allocation, be sure to leave at least 1-2 GB of memory for the OS.

Default Settings​

By default if left unset, DREMIO_MAX_HEAP_MEMORY_SIZE_MB and DREMIO_MAX_DIRECT_MEMORY_SIZE_MB are set as follows.

Coordinator Nodes

  1. If DREMIO_MAX_MEMORY_SIZE_MB is ≥18 GB, Heap is 16 GB and Direct uses the remaining.
  2. If DREMIO_MAX_MEMORY_SIZE_MB is <18 GB, Direct is 2 GB and Heap uses the remaining.

Executor Nodes

  1. If DREMIO_MAX_MEMORY_SIZE_MB is ≥32 GB, Heap is 8 GB and Direct uses the remaining.
  2. If DREMIO_MAX_MEMORY_SIZE_MB is ≥6 GB, Heap is 4 GB and Direct uses the remaining.
  3. Otherwise, Heap is 2 GB and Direct uses the remaining.
Note: Dremio requires DREMIO_MAX_MEMORY_SIZE_MB to be at least 4 GB.

Heap Monitor​

To prevent JVM failures due to a lack of heap memory, Dremio's heap monitor kills queries if the JVM's heap utilisation stays above the threshold of 85%, even after a full garbage collection. On an executor node, the heap monitor evaluates currently running queries and kills as many needed to release approximately 25% of the memory heap. If necessary, the heap monitor runs again to kill more queries. On a coordinator node, the heap monitor cancels all queries in the planning phase. The feature is enabled by default.

The query profile for a query killed by a heap monitor on an exector node contains the following error:

Query was cancelled because it exceeded the memory limits set by the administrator.

The query profile for a query killed on a coordinator node contains the following error:

Query canceled - out of memory, check the query profile for details

Dremio recommends trying the following suggestions, in order, if the heap monitor kills a query:

  1. Retry the query.
    If there are fewer queries in parallel during the second attempt, the query should succeed.
  2. Reduce query parallelism.
  3. Increase the heap memory.
  4. Increase the number of executor nodes.