Kafka Partitions

The main lever you can adjust to tune Kafka throughput is the number of partitions.

When you calculate the number of Kafka partitions, it is important to remember that it is the unit of parallelism. A topic with more partitions has higher throughput and higher latency than a topic with less. It is important to determine the correct number of partitions because too many will lead to unnecessarily higher latency while too few will not meet throughput expectations. This parameter is not directly configured via Metron but rather when manually creating the associated Kafka topics.