Kernel and OS tuning
The Cloudera Runtime platform depends on a tuned underlying host operating system (OS) for optimal performance. Cloudera suggests setting the vm.swappiness and transparent hugepage compaction kernel parameters.
For additional background information and suggested settings, see Optimizing Performance in Cloudera Runtime.
Cryptographic operations require entropy to ensure randomness. The Cloudera Security guide explains how to check available entropy and how to ensure sufficient entropy is available: Entropy Requirements.
ethtool -g eth0 Ring parameters for eth0: Pre-set maximums: RX: 4096 RX Mini: 0 RX Jumbo: 0 TX: 4096 Current hardware settings: RX: 256 RX Mini: 0 RX Jumbo: 0 TX: 256
ethtool –G [***INTERFACE***] rx [***NEWSIZE***]
ethtool –G [***INTERFACE***] tx [***NEWSIZE***]
Most modern enterprise-grade network adapters have several performance optimization features, such as offload capabilities and large segmentation offload, which reduce load on host CPU by handling these functions at the network interface level which can be explored as part of performance optimization initiatives. An iterative approach is recommended while applying a standard load generator such as the Terasuite benchmarks, to test the effect of enabling said features. Performance optimization parameters should not be applied indiscriminately, without thorough testing, and should only be applied based on genuine need.
The following parameters should be added to /etc/sysctl.conf to optimize various network behaviors:
- Disable TCP timestamps to improve CPU utilization (this is an optional parameter
and depends on your NIC
- Enable TCP sacks to improve
- Increase the maximum length of processor input
- Increase the TCP max and default buffer sizes using
net.core.rmem_max=4194304 net.core.wmem_max=4194304 net.core.rmem_default=4194304 net.core.wmem_default=4194304 net.core.optmem_max=4194304
- Increase memory thresholds to prevent packet
net.ipv4.tcp_rmem=4096 87380 4194304 net.ipv4.tcp_wmem=4096 65536 4194304