Disabling Transparent Hugepages (THP)

Most Linux platforms supported by Cloudera Runtime include a feature called transparent hugepages, which interacts poorly with Hadoop workloads and can seriously degrade performance.

Symptoms:

Symptom 1: High system CPU usage and performance degradation

System monitoring tools, such as top, show a large percentage of CPU usage under "system CPU." If system CPU usage consumes 30% or more of the total CPU capacity, your system might be experiencing this issue.

To check if the system enables transparent hugepages, run the following commands and verify the output:

$ cat defrag_file_pathname
$ cat enabled_file_pathname

[always] never means that transparent hugepages is enabled.
always [never] means that transparent hugepages is disabled.

Symptom 2: Server native memory leak and 100% CPU spin (OPSAPS-77888)

When you enable Transparent Huge Pages (THP) and set them to always on the cluster, a known issue with the Oracle Database driver (Oracle Instant Client) triggers severe memory instability by causing it to consume a large amount of memory.

The AnonHugePages memory allocation grows continuously, causing a server native memory leak. Consequently, the memory consumption of processes using this database driver, such as Cloudera Manager, keeps growing and escalates until it induces a 100% CPU spin, which eventually prevents all other system processes from functioning.

To disable Transparent Hugepages, perform the following steps on all cluster hosts:

To disable transparent hugepages on reboot, add the following commands to the /etc/rc.d/rc.local file on all cluster hosts:

RHEL/CentOS 7.x, 8.x, and 9.x:

echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

RHEL/CentOS 6.x

echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled

Ubuntu/Debian, OL, SLES:

echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled

Modify the permissions of the rc.local file:

chmod +x /etc/rc.d/rc.local

If your cluster hosts are using the GRUB bootloader, modify the GRUB configuration to disable THP:
1. Add the following line to the GRUB_CMDLINE_LINUX options in the /etc/default/grub file:
```
transparent_hugepage=never
```
2. Run the following command:
```
grub2-mkconfig -o /boot/grub2/grub.cfg
```
You can also disable transparent hugepages interactively (but remember this will not survive a reboot).
To disable transparent hugepages temporarily as root:
```
# echo 'never' > defrag_file_pathname
# echo 'never' > enabled_file_pathname 
```
To disable transparent hugepages temporarily using sudo:
```
$ sudo sh -c "echo 'never' > defrag_file_pathname"
$ sudo sh -c "echo 'never' > enabled_file_pathname" 
```