Cloudera Enterprise 5.x with EMC Isilon Scale-out Storage as DFS

In your compute nodes’ BIOS, set CPU to Performance mode for best performance.

Following CPUfreq governor types are available in RHEL 6 (check other OS-specific governors if you are not using CentOS or RHEL 6) –

Find the appropriate kernel modules for available on the system. And then use modprobe to add the driver needed.

# modprobe cpufreq_performance

Once a particular governor is loaded into the kernel, it can be enabled by using –

# cpupower frequency-set –governor cpufreq_performance

The available drivers can be found in the /lib/modules/<kernel version>/kernel/arch/<architecture>/kernel/cpu/cpufreq/ directory

/lib/modules/2.6.32-358.14.1.el6.centos.plus.x86_64/kernel/arch/x86/kernel/cpu/cpufreq

[root@ip-10-5-152-126 cpufreq]# ls

acpi-cpufreq.ko mperf.ko p4-clockmod.ko pcc-cpufreq.ko powernow-k8.ko speedstep-lib.ko

[root@ip-10-5-152-126 cpufreq]#

If the necessary cpufreq drivers are not available, you can get them from the /lib/modules/<kernel version>/kernel/drivers/cpufreq –

[root@ip-10-5-152-126 cpufreq]# cd /lib/modules/2.6.32-358.14.1.el6.centos.plus.x86_64/kernel/drivers/cpufreq

[root@ip-10-5-152-126 cpufreq]# ls

cpufreq_conservative.ko cpufreq_ondemand.ko cpufreq_powersave.ko cpufreq_stats.ko freq_table.ko

[root@ip-10-5-152-126 cpufreq]#

NOTE: The “uname –r” command will give you the kernel version.

The “cpupower” utility is provided by the cpupowerutils package.

If you don’t have it installed, you can set the tunables in /sys/devices/system/cpu/<cpu id>/cpufreq/

Minimize anonymous page faults by setting vm.swappiness = 1 , thereby freeing from page cache before “Swapping” application pages (this reduces the OOM-killer invocation)

Edit /etc/sysctl.conf in editor of choice and add following line –

 vm.swappiness=1

After that, run –

# sysctl –p
# sysctl –a|grep "vm.swappiness"

echo "never" >
/sys/kernel/mm/redhat_transparent_hugepage/enabled[

echo "never" > /sys/kernel/mm/redhat_transparent_hugepage/defrag

Add these commands to /etc/rc.local to ensure that transparent hugepage compaction and defragmentation remain disabled across reboots.

Check what features are available with your NIC by using “ethtool” –

$ sudo ethtool -k
Features for eth0:
rx-checksumming: on
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: off

There are various offload capabilities modern NICs (and especially high-performance NICs) have and it is always recommended to enable them.

A few features such as tcp-segmentation-offload (TSO), scatter-gather (SG) and generic-segmentation-offload (GSO) are usually good features to enable (if not enabled by default).

Check existing ring buffer sizes by running

$ ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX:                4096
RX Mini:        0
RX Jumbo:        0
TX:                4096
Current hardware settings:
RX:                256
RX Mini:        0
RX Jumbo:        0
TX:                256

After checking the pre-set maximum values, and the current hardware settings, the following command can be used to resize the ring buffers –

# ethtool –G <interface> rx <newsize>

or

# ethtool –G <interface> tx <newsize>

NOTE: The ring buffer sizes will depend on the network topology to a certain degree and might need to be tuned depending on the nature of the workload. We have seen for 10G NICs, setting the RX and TX buffers to maximum is reasonable. This will of course have to be tuned based on the network architecture and type of traffic.

Disable “atime” from the data disks (and root fs for that matter) - use “noatime” option during mounting the FS

In the /etc/fstab file ensure that the appropriate filesystems have the “noatime” mount option specified -

LABEL=ROOT /         ext4    noatime        0 0

FS creation - enable journal mode, reduce superuser block reservation from 5% to 1% for root (using the -m1 option), the sparse_super,dir_index,extent options (minimize number of super block backups and use b-tree indexes for directory trees, extent based allocations)

# mkfs –t ext4 –m1 –O sparse_super,dir_index,extent,has_journal /dev/sdb1