Checking Mirroring Progress

You can use Kafka's Consumer Offset Checker command-line tool to assess how well your mirror is keeping up with the source cluster. The Consumer Offset Checker checks the number of messages read and written, and reports the lag for each consumer in a specified consumer group.

The following command runs the Consumer Offset Checker for group KafkaMirror, topic test-topic. The --zkconnect argument points to the ZooKeeper host and port on the source cluster.

/usr/hdp/current/kafka/bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror --zkconnect source-cluster-zookeeper:2181 --topic test-topic

Group        Topic          Pid Offset     logSize     Lag     Owner
KafkaMirror  test-topic     0   5          5           0       none
KafkaMirror  test-topic     1   3          4           1       none
KafkaMirror  test-topic     2   6          9           3       none

Table 1. Consumer Offset Checker Options
`--group`	(Required) Specifies the consumer group.
`--zkconnect`	Specifies the ZooKeeper connect string. The default is `localhost:2181`.
`--broker-info`	Lists broker information
`--help`	Lists offset checker options.
`--topic`	Specifies a comma-separated list of consumer topics. If you do not specify a topic, the offset checker will display information for all topics under the given consumer group.