Most Linux platforms supported by Cloudera Runtime include a
feature called transparent hugepages, which interacts poorly
with Hadoop workloads and can seriously degrade performance.
Symptom:
top and other system monitoring tools show a large percentage of the CPU
usage classified as "system CPU". If system CPU usage is 30% or more of the total CPU usage,
your system may be experiencing this issue.
To see whether transparent hugepages are enabled, run the following commands and check the
output:
$ cat defrag_file_pathname
$ cat enabled_file_pathname
-
[always] never means that transparent hugepages is enabled.
-
always [never] means that transparent hugepages is disabled.
To disable Transparent Hugepages, perform the following steps on all cluster hosts:
-
To disable transparent hugepages on reboot, add the following commands to the
/etc/rc.d/rc.local file on all cluster hosts:
-
RHEL/CentOS 7.x, 8.x, and 9.x:
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
-
RHEL/CentOS 6.x
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
-
Ubuntu/Debian, OL, and SLES:
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
Modify the permissions of the
rc.local
file:
chmod +x /etc/rc.d/rc.local
-
If your cluster hosts are using the GRUB bootloader, modify the GRUB configuration to
disable THP:
-
Add the following line to the
GRUB_CMDLINE_LINUX options in the
/etc/default/grub file:
transparent_hugepage=never
-
Run the following command:
grub2-mkconfig -o /boot/grub2/grub.cfg
- Disable the tuned service, as described above.
You can also disable transparent hugepages interactively (but remember this will not
survive a reboot).
To disable transparent hugepages temporarily as root:
# echo 'never' > defrag_file_pathname
# echo 'never' > enabled_file_pathname
To disable transparent hugepages temporarily using sudo:
$ sudo sh -c "echo 'never' > defrag_file_pathname"
$ sudo sh -c "echo 'never' > enabled_file_pathname"
