Memory is configured via the /etc/dremio/dremio-env file. When this configuration file is changed from the default, it must be updated on all nodes of interest.
[info] Note: In addition to memory, this configuration file is used to set Java options and log directories.
Maximum Memory (recommended to configure)
Maximum memory size allows Dremio to automatically determine the best allocation between HEAP and DIRECT memory depending on the node type. To change the default maximum memory, modify the following line:
Alternatively, you can set HEAP and DIRECT memory instead of maximum memory. The maximum HEAP and DIRECT memory can be set separately.
- If both are specified, then the maximum memory option is ignored.
- If only one is configured, Dremio automatically determines the other, based on leftover memory.
Heap memory is used for running Dremio server. To change the default maximum heap memory, modify the following line:
Direct memory is used for query execution. To change the default maximum direct memory, modify the following line:
If you see queries failing on an instance that's running out of memory, increasing the amount of memory that Dremio is able to consume may solve your problem.
[warning] Warning: For the
DREMIO_MAX_DIRECT_MEMORY_SIZE_MBallocation, be sure to leave at least 1-2 GB of memory for the OS.
By default if left unset,
DREMIO_MAX_DIRECT_MEMORY_SIZE_MB are set as follows.
DREMIO_MAX_MEMORY_SIZE_MBis ≥18 GB, Heap is 16 GB and Direct uses the remaining.
DREMIO_MAX_MEMORY_SIZE_MBis <18 GB, Direct is 2 GB and Heap uses the remaining.
DREMIO_MAX_MEMORY_SIZE_MBis ≥32 GB, Heap is 8 GB and Direct uses the remaining.
DREMIO_MAX_MEMORY_SIZE_MBis ≥6 GB, Heap is 4 GB and Direct uses the remaining.
- Otherwise, Heap is 2 GB and Direct uses the remaining.
Note: Dremio requires DREMIO_MAX_MEMORY_SIZE_MB to be at least 4 GB.
To prevent JVM failures due to a lack of heap memory, Dremio's heap monitor kills queries if the JVM's heap utilisation stays above the threshold of 85%, even after a full garbage collection. On an executor node, the heap monitor evaluates currently running queries and kills as many needed to release approximately 25% of the memory heap. If necessary, the heap monitor runs again to kill more queries. On a coordinator node, the heap monitor cancels all queries in the planning phase. The feature is enabled by default.
The query profile for a query killed by a heap monitor on an exector node contains the following error:
Query was cancelled because it exceeded the memory limits set by the administrator.
The query profile for a query killed on a coordinator node contains the following error:
Query canceled - out of memory, check the query profile for details
Dremio recommends trying the following suggestions, in order, if the heap monitor kills a query:
- Retry the query.
If there are fewer queries in parallel during the second attempt, the query should succeed.
- Reduce query parallelism.
- Increase the heap memory.
- Increase the number of executor nodes.