Mirroring data between Kafka clusters
Also available as:
PDF

Checking Mirroring Progress

You can use Kafka's Consumer Offset Checker command-line tool to assess how well your mirror is keeping up with the source cluster. The Consumer Offset Checker checks the number of messages read and written, and reports the lag for each consumer in a specified consumer group.

The following command runs the Consumer Offset Checker for group KafkaMirror, topic test-topic. The --zkconnect argument points to the ZooKeeper host and port on the source cluster.

/usr/hdp/current/kafka/bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror --zkconnect source-cluster-zookeeper:2181 --topic test-topic

Group        Topic          Pid Offset     logSize     Lag     Owner
KafkaMirror  test-topic     0   5          5           0       none
KafkaMirror  test-topic     1   3          4           1       none
KafkaMirror  test-topic     2   6          9           3       none
Table 1. Consumer Offset Checker Options
--group (Required) Specifies the consumer group.
--zkconnect Specifies the ZooKeeper connect string. The default is localhost:2181.
--broker-info Lists broker information
--help Lists offset checker options.
--topic Specifies a comma-separated list of consumer topics. If you do not specify a topic, the offset checker will display information for all topics under the given consumer group.