Tuning JVM Garbage Collection

When using OpenJDK 11, Cloudera Manager and most CDH services use G1GC as the default method of garbage collection. (Java 8 used "ConcurrentMarkSweep" (CMS) for garbage collection.) When using G1GC, the pauses for garbage collection are shorter, so components will usually be more responsive, but they are more sensitive to overcommitted memory usage. You should monitor memory usage to determine whether memory is overcommitted.
Cloudera Manager alerts you when memory is overcommitted on cluster hosts. To view these alerts and adjust the allocations:
  1. Log in to the Cloudera Manager Admin Console
  2. Go to Home > Configuration > Configuration Issues.
  3. Look for entries labeled Memory Overcommit Validation Threshold and note the hostname of the affected host.
  4. Go to Hosts > All Hosts and click on the affected host.
  5. Click the Resources tab.
  6. Scroll down to the Memory section.

    A list of roles instances and their memory allocations are displayed. The Description column displays the configuration property name where the memory allocation can be set.

  7. To adjust the memory allocation, search for the configuration property and adjust the value to reduce the overcommitment of memory. You may need to move some roles to other hosts if there is not sufficient memory for the roles running on the host.
  8. After making any changes, Cloudera Manager will indicate that the service has a stale configuration and prompt you to restart the service.

You may also need to adjust the Java options used to start Java processes. You can add Java startup options using Cloudera Manager configuration properties that are available for all service roles. Cloudera has provided default arguments for some of the services where they are needed. You can add to these, or completely override all of the provided Java options. For more information on configuring G1GC. see The OpenJDK documentation.

If default options are provided, the role configuration specifies a single value, {{JAVA_GC_ARGS}}. This value is a placeholder for the default Java Garbage Collection options provided with Cloudera Manager and CDH.

To modify Java options:
  1. Log in to the Cloudera Manager Admin Console.
  2. Go to the service where you want to modify the options.
  3. Select the Configuration tab.
  4. Enter "Java" in the search box.
  5. Locate the Java Configuration Options property named for the role you want to modify. For example, in the HDFS service, you will see parameters like Java Configuration Options for DataNode and Java Configuration Options for JournalNode.
  6. To add to the Java options, enter additional options before or after the {{JAVA_GC_ARGS}} placeholder, separated by spaces. For example: 
    {{JAVA_GC_ARGS}} -XX:MaxPermSize=512M
  7. To replace the default Java options, delete the {{JAVA_GC_ARGS}} placeholder and replace it with one or more Java options, separated by spaces.
  8. The service will now have a stale configuration and must be restarted. See Restarting a Service.
Default Java Options
Service and Role Default Java 8 Options Default Java 11 Options
  • HDFS DataNode
  • HDFS NameNode
  • HDFS Secondary NameNode
 
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled
 
-XX:+UseConcMarkSweepGC 
-XX:CMSInitiatingOccupancyFraction=70 
-XX:+CMSParallelRemarkEnabled
  • Hive Metastore Server
  • HiveServer 2
  • WebHCat Server
 
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled
 None, G1GC is enabled by default.
  • HBase REST Server
  • HBase Thrift Server
  • HBase Master
  • HBase RegionServer
 
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled
 None, G1GC is enabled by default.
  • HBase Region Server
 
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled
-verbose:gc -XX:+PrintGCDetails 
-XX:+PrintGCDateStamps
 
-verbose:gc -Xlog:gc
  • MapReduce JobTracker
  • MapReduce TaskTracker
 
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled
 None, G1GC is enabled by default.
  • Solr Server
 
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled
 None, G1GC is enabled by default.
  • YARN JobHistory Server
  • YARN NodeManager
  • YARN Resource Manager
 
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled
-Dlibrary.leveldbjni.path={{CMF_CONF_DIR}}
 
-Dlibrary.leveldbjni.path={{CMF_CONF_DIR}}