Optimizing Performance in Cloudera Runtime

This section provides solutions to some performance problems, and describes configuration best practices.

important

Work with your network administrators and hardware vendors to ensure that you have the proper NIC firmware, drivers, and configurations in place and that your network performs properly. Cloudera recognizes that network setup and upgrade are challenging problems, and will do its best to share useful experiences.

Disabling Transparent Hugepages (THP)
Most Linux platforms supported by Cloudera Runtime include a feature called transparent hugepages, which interacts poorly with Hadoop workloads and can seriously degrade performance.
Setting the vm.swappiness Linux Kernel Parameter
The Linux kernel parameter, vm.swappiness, is a value from 0-100 that controls the swapping of application data (as anonymous pages) from physical memory to virtual memory on disk. You can set the value of the vm.swappiness parameter for minimum swapping.
Improving Performance in Shuffle Handler and IFile Reader
The MapReduce shuffle handler and IFile reader use native Linux calls, (posix_fadvise(2) and sync_data_range), on Linux systems with Hadoop native libraries installed.
Tips and Best Practices for Jobs
This section describes changes you can make at the job level.
Decrease Reserve Space
By default, the ext3 and ext4 filesystems reserve 5% space for use by the root user. This reserved space counts as Non DFS Used.

Parent topic: Performance Management