Cloudera Docs

Kafka clients and ZooKeeper

Learn more about the differences between the old and new model for storing consumer offsets.

The default consumer model provides the metadata for offsets in the Kafka cluster. There is a topic named __consumer_offsets that the Kafka consumers write their offsets to.

Figure 1. Kafka Consumer Dependencies

In releases before version 2.0 of CDK Powered by Apache Kafka, the same metadata was located in ZooKeeper. The new model removes the dependency and load from Zookeeper. In the old approach:

  • The consumers save their offsets in a "consumer metadata" section of ZooKeeper.
  • With most Kafka setups, there are often a large number of Kafka consumers. The resulting client load on ZooKeeper can be significant, therefore this solution is discouraged.
Figure 2. Kafka Consumer Dependencies (Old Approach)