Setting User Limits for Kafka
Kafka opens many files at the same time. The default setting of 1024 for the maximum number of open files on most Unix-like systems is insufficient. Any significant load can result in failures and cause error messages such as java.io.IOException...(Too many open files) to be logged in the Kafka or HDFS log files. You might also notice errors such as this:
ERROR Error in acceptor (kafka.network.Acceptor) java.io.IOException: Too many open files
Cloudera recommends setting the value to a relatively high starting point, such as 32,768.
You can monitor the number of file descriptors in use on the Kafka Broker dashboard. In Cloudera Manager:
- Go to the Kafka service.
- Select a Kafka Broker.
- Open File Descriptors chart. and scroll down to the