Kafka clients and ZooKeeper
Learn more about the differences between the old and new model for storing consumer offsets.
The default consumer model provides the metadata for offsets in the Kafka cluster. There is a
__consumer_offsets that the Kafka consumers write their offsets
In releases before version 2.0 of CDK Powered by Apache Kafka, the same metadata was located in ZooKeeper. The new model removes the dependency and load from Zookeeper. In the old approach:
- The consumers save their offsets in a "consumer metadata" section of ZooKeeper.
- With most Kafka setups, there are often a large number of Kafka consumers. The resulting client load on ZooKeeper can be significant, therefore this solution is discouraged.