Tooling
You can use the following Storm and Kafka tools to monitor your topologies.
Kafka
Consumer group offset lag viewer
Kafka Manager - A GUI tool to facilitate creating, modifying, and managing your Kafka topics
Console consumer - useful for quickly verifying topic contents
Storm
The Storm user interface is a very useful tool for monitoring your topologies. For more information on the Storm user interface, see Reading and Understanding the Storm UI.
Example: Viewing Kafka Offset Lags
You can use the Kafka consumer group offset lag viewer to monitor the delta calculations between the current and end offset for a partition.
Set up some environment variables.
export BROKERLIST your broker comma-delimated list of host:ports> export ZOOKEEPER your zookeeper comma-delimated list of host:ports> export KAFKA_HOME kafka home dir> export METRON_HOME your metron home> export HDP_HOME your HDP home>
If you have Kerberos enabled, set up the security protocol.
$ cat /tmp/consumergroup.config security.protocol=SASL_PLAINTEXT
Enter the following command to display a table containing offsets for all partitions and consumers associated with a running topology's consumer group:
${KAFKA_HOME}/bin/kafka-consumer-groups.sh \ --command-config=/tmp/consumergroup.config \ --describe \ --group enrichments \ --bootstrap-server $BROKERLIST \ --new-consumer
The command displays the following table:
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER enrichments enrichments 9 29746066 29746067 1 consumer-2_/xxx.xxx.xxx.xxx enrichments enrichments 3 29754325 29754326 1 consumer-1_/xxx.xxx.xxx.xxx enrichments enrichments 43 29754331 29754332 1 consumer-6_/xxx.xxx.xxx.xxx ...
Note Output displays only when the topology is running because the consumer groups only exist while consumers in the spouts are up and running.
The LAG column lists the current delta calculation between the current and end offset for the partition. The column value indicates how close you are to keeping up with incoming data. It also indicates whether there are any problems with specific consumers getting stuck.
To watch the offsets and lags change over time, add a watch command and set the refresh rate to 10 seconds:
watch -n 10 -d ${KAFKA_HOME}/bin/kafka-consumer-groups.sh \ --command-config=/tmp/consumergroup.config \ --describe \ --group enrichments \ --bootstrap-server $BROKERLIST \ --new-consumer
The watch command runs every 10 seconds and refreshes the screen with new information. The command also highlights the differences from the current output and the last output screens.