View Kafka Offset Lags Example
You can use the Kafka consumer group offset lag viewer to monitor the delta calculations between the current and end offset for a partition.
-
Set up some environment variables.
export BROKERLIST your broker comma-delimated list of host:ports> export ZOOKEEPER your zookeeper comma-delimated list of host:ports> export KAFKA_HOME kafka home dir> export METRON_HOME your metron home> export HDP_HOME your HDP home>
-
If you have Kerberos enabled, set up the security protocol.
$ cat /tmp/consumergroup.config security.protocol=SASL_PLAINTEXT
-
Enter the following command to display a table containing offsets for all partitions
and consumers associated with a running topology's consumer group:
${KAFKA_HOME}/bin/kafka-consumer-groups.sh \ --command-config=/tmp/consumergroup.config \ --describe \ --group enrichments \ --bootstrap-server $BROKERLIST \ --new-consumer
The command displays the following table:GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER enrichments enrichments 9 29746066 29746067 1 consumer-2_/xxx.xxx.xxx.xxx enrichments enrichments 3 29754325 29754326 1 consumer-1_/xxx.xxx.xxx.xxx enrichments enrichments 43 29754331 29754332 1 consumer-6_/xxx.xxx.xxx.xxx ...
The LAG column lists the current delta calculation between the current and end offset for the partition. The column value indicates how close you are to keeping up with incoming data. It also indicates whether there are any problems with specific consumers getting stuck. -
To watch the offsets and lags change over time, add a watch command and set the
refresh rate to 10 seconds:
watch -n 10 -d ${KAFKA_HOME}/bin/kafka-consumer-groups.sh \ --command-config=/tmp/consumergroup.config \ --describe \ --group enrichments \ --bootstrap-server $BROKERLIST \ --new-consumer
The watch command runs every 10 seconds and refreshes the screen with new information. The command also highlights the differences from the current output and the last output screens.