A collection of basic recommendations for Kafka clusters.
The simplest recommendation for running Kafka with maximum performance is to have dedicated hosts for the Kafka brokers and a dedicated ZooKeeper cluster for the Kafka cluster. If that is not an option, consider these additional guidelines for resource sharing with the Kafka cluster:
- Running in VMs
- It is common practice in modern data centers to run processes in virtual machines. This generally allows for better sharing of resources. Kafka is sufficiently sensitive to I/O throughput that VMs interfere with the regular operation of brokers. For this reason, it is generally not recommended to run Kafka in VMs. However, if you are running Kafka in a virtual environment you will need to rely on your VM vendor for help with optimizing Kafka performance.
- Do not run other processes with Brokers or ZooKeeper
- Due to I/O contention with other processes, it is generally recommended to avoid running other such processes on the same hosts as Kafka brokers.
- Keep the Kafka-ZooKeeper Connection Stable
- Kafka relies heavily on having a stable ZooKeeper connection. Putting an unreliable
network between Kafka and ZooKeeper will appear as if ZooKeeper is offline to Kafka.
Examples of unreliable networks include:
- Do not put Kafka/ZooKeeper nodes on separated networks
- Do not put Kafka/ZooKeeper nodes on the same network with other high network loads