File descriptor limits
Learn more about file descriptor requirements.
As Kafka works with many log segment files and network connections, the
Process File Descriptors setting may need to be increased in some cases in
production deployments, if a broker hosts many partitions. For example, a Kafka broker needs
at least the following number of file descriptors to just track log segment files:
(number of partitions)*(partition size / segment size)
The broker needs additional file descriptors to communicate via network sockets with external parties (such as clients, other brokers, Zookeeper, Sentry, and Kerberos).
Maximum Process File Descriptors setting can be monitored in Cloudera
Manager and increased if usage requires a larger value than the default
ulimit (often 64K). It should be reviewed for use case suitability.
- To review FD limit currently set for a running Kafka broker, run
cat /proc/KAFKA_BROKER_PID/limits, and look for
Max open files.
- To see open file descriptors, run:
lsof -p KAFKA_BROKER_PID