Adjust the Maximum Number of Open Files and Processes

In a secure cluster, if the DataNodes are started as the root user, JSVC downgrades the processing using setuid to hdfs. However, the ulimit is based on the ulimit of the root user, and the default ulimit values assigned to the root user for the maximum number of open files and processes may be too low for a secure cluster. This can result in a “Too Many Open Files” exception when the DataNodes are started.

Therefore, when configuring a secure cluster you should increase the following root ulimit values:

To set system-wide ulimits to these values, log in as root and add the following lines to the /etc/security/limits.conf file on every host in your cluster:

* - nofile 65536
* - nproc 65536

To set only the root user ulimits to these values, log in as root and add the following lines to the /etc/security/limits.conf file.

root - nofile 65536
root - nproc 65536

You can use the ulimit -a command to view the current settings:

[root@node-1 /]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 14874
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 14874
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

You can also use the ulimit command to dynamically set these limits until the next reboot. This method sets a temporary value that will revert to the settings in the /etc/security/limits.conf file after the next reboot, but it is useful for experimenting with limit settings. For example:

[root@node-1 /]# ulimit -n 65536

The updated value can then be displayed:

[root@node-1 /]# ulimit -n
65536