Memory is configured via the /etc/dremio/dremio-env file. When this configuration file is changed from the default, it must be updated on all nodes of interest.
[info] Note: In addition to memory, this configuration file is used to set Java options and log directories.
Maximum Memory (recommended to configure)
Maximum memory size allows Dremio to automatically determine the best allocation between HEAP and DIRECT memory depending on the node type. To change the default maximum memory, modify the following line:
Alternatively, you can set HEAP and DIRECT memory instead of maximum memory. The maximum HEAP and DIRECT memory can be set separately.
- If both are specified, then the maximum memory option is ignored.
- If only one is configured, Dremio automatically determines the other, based on leftover memory.
Heap memory is used for running Dremio server. To change the default maximum heap memory, modify the following line:
Direct memory is used for query execution. To change the default maximum direct memory, modify the following line:
If you see queries failing on an instance that's running out of memory, increasing the amount of memory that Dremio is able to consume may solve your problem.
[warning] Warning: For the
DREMIO_MAX_DIRECT_MEMORY_SIZE_MBallocation, be sure to leave at least 1-2 GB of memory for the OS.
By default if left unset,
DREMIO_MAX_DIRECT_MEMORY_SIZE_MB are set as follows.
DREMIO_MAX_MEMORY_SIZE_MBis ≥18 GB, Heap is 16 GB and Direct uses the remaining.
DREMIO_MAX_MEMORY_SIZE_MBis <18 GB, Direct is 2 GB and Heap uses the remaining.
DREMIO_MAX_MEMORY_SIZE_MBis ≥32 GB, Heap is 8 GB and Direct uses the remaining.
DREMIO_MAX_MEMORY_SIZE_MBis ≥6 GB, Heap is 4 GB and Direct uses the remaining.
- Otherwise, Heap is 2 GB and Direct uses the remaining.
Note: Dremio requires DREMIO_MAX_MEMORY_SIZE_MB to be at least 4 GB.