Disabling Transparent Hugepages (THP)

Most Linux platforms supported by Cloudera Runtime include a feature called transparent hugepages, which interacts poorly with Hadoop workloads and can seriously degrade performance.

Symptom: top and other system monitoring tools show a large percentage of the CPU usage classified as "system CPU". If system CPU usage is 30% or more of the total CPU usage, your system may be experiencing this issue.

To see whether transparent hugepages are enabled, run the following commands and check the output:

$ cat defrag_file_pathname
$ cat enabled_file_pathname
  • [always] never means that transparent hugepages is enabled.
  • always [never] means that transparent hugepages is disabled.

To disable Transparent Hugepages, perform the following steps on all cluster hosts:

  1. (Required for hosts running RHEL/CentOS 7.x.) To disable transparent hugepages on reboot, add the following commands to the /etc/rc.d/rc.local file on all cluster hosts:
    • RHEL/CentOS 7.x:
      echo never > /sys/kernel/mm/transparent_hugepage/enabled
      echo never > /sys/kernel/mm/transparent_hugepage/defrag
    • RHEL/CentOS 6.x
      echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
      echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
    • Ubuntu/Debian, OL, SLES:
      echo never > /sys/kernel/mm/transparent_hugepage/defrag
      echo never > /sys/kernel/mm/transparent_hugepage/enabled
    Modify the permissions of the rc.local file:
    chmod +x /etc/rc.d/rc.local
  2. If your cluster hosts are running RHEL/CentOS 7.x, modify the GRUB configuration to disable THP:
    1. Add the following line to the GRUB_CMDLINE_LINUX options in the /etc/default/grub file:
    2. Run the following command:
      grub2-mkconfig -o /boot/grub2/grub.cfg
  3. Disable the tuned service, as described above.

    You can also disable transparent hugepages interactively (but remember this will not survive a reboot).

    To disable transparent hugepages temporarily as root:

    # echo 'never' > defrag_file_pathname
    # echo 'never' > enabled_file_pathname 

    To disable transparent hugepages temporarily using sudo:

    $ sudo sh -c "echo 'never' > defrag_file_pathname"
    $ sudo sh -c "echo 'never' > enabled_file_pathname"