G1 Garbage Collection Setup

This topic describes how to implement Garbage-First (G1) garbage collection (GC). G1 is a server-style garbage collector for multi-processor machines with large memories.

Scenario: Dremio executors are not deployed on Yarn

To implement G1 garbage collection and heap dump flags when Dremio executors are not on Yarn,
add the following property to the dremio-env file on all Dremio coordinator and executor nodes.

DREMIO_JAVA_SERVER_EXTRA_OPTS="-XX:+UseG1GC -XX:G1HeapRegionSize=32M -XX:MaxGCPauseMillis=500 -XX:InitiatingHeapOccupancyPercent=25"

By default, the GC logs and heap dumps are stored under the Dremio log folder. The following is an example: 501 3479 1 0 Mon01PM ttys001 36:38.94 /usr/bin/java -Djava.util.logging.config.class=org.slf4j.bridge.SLF4JBridgeHandler -Djava.library.path=/Users/dremio/Downloads/dremio-enterprise-3.2.0-201905102005330382-0598733/lib -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/Users/dremio/Downloads/dremio-enterprise-3.2.0-201905102005330382-0598733/log/server.gc -Ddremio.log.path=/Users/dremio/Downloads/dremio-enterprise-3.2.0-201905102005330382-0598733/log -Xmx4096m -XX:MaxDirectMemorySize=8192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/Users/dremio/Downloads/dremio-enterprise-3.2.0-201905102005330382-0598733/log -Dio.netty.maxDirectMemory=0 -DMAPR_IMPALA_RA_THROTTLE -DMAPR_MAX_RA_STREAMS=400 -XX:+UseG1GC -XX:G1HeapRegionSize=32M -XX:MaxGCPauseMillis=500 -XX:InitiatingHeapOccupancyPercent=25 -cp /Users/dremio/Downloads/dremio-enterprise-3.2.0-201905102005330382-0598733/conf:/Users/dremio/Downloads/dremio-enterprise-3.2.0-201905102005330382-0598733/jars/*:/Users/dremio/Downloads/dremio-enterprise-3.2.0-201905102005330382-0598733/jars/ext/*:/Users/dremio/Downloads/dremio-enterprise-3.2.0-201905102005330382-0598733/jars/3rdparty/* com.dremio.dac.daemon.DremioDaemon

Scenario: Dremio executors are deployed on Yarn

To implement G1 garbage collection and heap dump flags when Dremio executors are deployed on Yarn,
add the following properties to the data source via the Dremio UI > Advanced Options > Connection Properties.

Property Type Name Value
JAVA provisioning.yarn.heapsize 8192
SYSTEM -XX:+UseG1GC
SYSTEM -XX:+HeapDumpOnOutOfMemoryError
SYSTEM -XX:HeapDumpPath <path>
SYSTEM -XX:+PrintGCDetails
SYSTEM -XX:+PrintGCTimeStamps
SYSTEM -XX:+PrintGCDateStamps
SYSTEM -XX:ErrorFile <path>/hs_err_pid%p.log
SYSTEM -XX:G1HeapRegionSize 32M
SYSTEM -XX:MaxGCPauseMillis 500
SYSTEM -XX:InitiatingHeapOccupancyPercent 25
SYSTEM -Xloggc:/<path-to-gc-logs>/gc.log-date +'%Y%m%d%H%M' <path>/hs_err_pid%p.log
SYSTEM -XX:ErrorFile

[info] For Kubernetes-based deployments, garbage collection is done via the Helm charts.


results matching ""

    No results matching ""