Cloudera Docs

HeapDumpPath (/tmp) in Hive data nodes gets full due to .hprof files

Condition

The HeapDumpPath (/tmp) in Hive nodes often gets full because of the .hprof files that are generated. Manually deleting these files to free up space does not always help because the files are generated every time there is an exception. Learn how to resolve this issue by modifying the relevant Tez properties.

Cause

A .hprof file is created when a process exits with an Out Of Memory exception, and the file captures the state of Java Virtual Machine (JVM) when it crashes.

The default value for HeapDumpPath in the Tez Application Master and Tez Task Command Line Options properties is set to /tmp and therefore, Tez saves all the heap dump files in this location.

If you do not want the heap dump files to be saved in /tmp, you can either set the HeapDumpPath value to a different location on the node or choose not to capture heap dump (.hprof) files when a process exits.

Solution

  1. Log in to Cloudera Manager as an administrator.
  2. Go to Clusters > Tez > Configuration and search for the 'Tez Application Master Command Line Options' and 'Tez Task Command Line Options' properties.

    Tez properties to manage HeapDumpPath
  3. Choose one of the following solutions:
    If you want to...Then...
    Choose a different location in the Hive node to capture heap dumps Modify the value of XX:HeapDumpPath in the above properties from /tmp to a different location, for example, /var/tmp.
    Disable automatic capture of heap dumps Remove -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp parameters from the above properties.
  4. Click Save Changes.
  5. Restart the Tez service.