Configuring Memory

Memory is configured via the /etc/dremio/dremio-env file. When this configuration file is changed from the default, it must be updated on all nodes of interest.

note

In addition to memory, this configuration file is used to set Java options and log directories.

Configuring the Maximum Memory Size, Heap Memory, and Direct Memory

During installation, Dremio Software automatically determines the amount of system memory available and allocates it between heap and direct memory, depending on the Dremio node type.

As you continually monitor the performance of your query workloads and adjust your Dremio configuration, you might want to adjust the amount of memory allocated on your nodes. Consult with Dremio Support before making adjustments. Based on your workloads, Dremio Support might advise you to adjust the maximum memory available using the DREMIO_MAX_MEMORY_SIZE_MB parameter, or to configure heap and direct memory directly instead of maximum memory.

• If you configure both heap and direct memory, then the DREMIO_MAX_MEMORY_SIZE_MB parameter is ignored.
• If you configure only heap memory or direct memory, Dremio Software automatically configures the other, according to the amount of leftover memory.

Heap and Direct Memory

During a consultation, Dremio Support may advise you to adjust the maximum allowed size of the heap memory, the maximum allowed size of the direct memory, or both.

To change the maximum size of the heap memory on a node, modify the following line:

DREMIO_MAX_HEAP_MEMORY_SIZE_MB=4096

To change the maximum size of the direct memory, modify the following line:

DREMIO_MAX_DIRECT_MEMORY_SIZE_MB=8192

caution

For the DREMIO_MAX_DIRECT_MEMORY_SIZE_MB allocation, be sure to leave at least 1-2 GB of memory for the OS.

Default Settings

By default if left unset, DREMIO_MAX_HEAP_MEMORY_SIZE_MB and DREMIO_MAX_DIRECT_MEMORY_SIZE_MB are set as follows.

Coordinator Nodes

1. If DREMIO_MAX_MEMORY_SIZE_MB is ≥18 GB, Heap is 16 GB and Direct uses the remaining.
2. If DREMIO_MAX_MEMORY_SIZE_MB is <18 GB, Direct is 2 GB and Heap uses the remaining.

Executor Nodes

1. If DREMIO_MAX_MEMORY_SIZE_MB is ≥32 GB, Heap is 8 GB and Direct uses the remaining.
2. If DREMIO_MAX_MEMORY_SIZE_MB is ≥6 GB, Heap is 4 GB and Direct uses the remaining.
3. Otherwise, Heap is 2 GB and Direct uses the remaining.

Heap Monitor

To prevent JVM failures due to a lack of heap memory, Dremio's heap monitor kills queries if the JVM's heap utilisation stays above the threshold of 85%, even after a full garbage collection. On an executor node, the heap monitor evaluates currently running queries and kills as many needed to release approximately 25% of the memory heap. If necessary, the heap monitor runs again to kill more queries. On a coordinator node, the heap monitor cancels all queries in the planning phase. The feature is enabled by default.

The query profile for a query killed by a heap monitor on an executor node contains the following error:

Query was cancelled because it exceeded the memory limits set by the administrator.

The query profile for a query killed on a coordinator node contains the following error:

Query canceled - out of memory, check the query profile for details

Dremio recommends trying the following suggestions, in order, if the heap monitor kills a query:

1. Retry the query.
   If there are fewer queries in parallel during the second attempt, the query should succeed.
2. Reduce query parallelism.
3. Increase the heap memory.
4. Increase the number of executor nodes.