Monitoring Kafka activity in Streams Messaging Manager
You can use Streams Messaging Manager (SMM) to monitor Kafka activity. To do this, you run two SSH sessions, one for producing data to a Kafka topic and the other to consume the data from the Kafka topic. Then you go to the SMM UI from the Management Console to track the producer and consumer activity.
Now that you know how to configure a Kafka client to connect to the Kafka cluster in
CDP, generate some activity and monitor it using SMM. To do this, you collect
machine usage data from the broker host using the
and stream that data into the
For detailed and comprehensive information about SMM, see Streams Messaging Manager.
- Start by opening two SSH connections to the same broker host.
On the first session, run the following command.
BROKER_HOST=$(hostname -f) vmstat 1 1000 | kafka-console-producer \ --broker-list $BROKER_HOST:9093 \ --producer.config client-ldap.properties \ --topic machine-data
This command runs
vmstatand sends each line from its output as one message every second to the
This example assumes that you are using LDAP authentication but you can use the authentication method of your preference.
Leave the session running.
On the second SSH session, run the following command to consume the data:
BROKER_HOST=$(hostname -f) kafka-console-consumer \ --bootstrap-server $BROKER_HOST:9093 \ --consumer.config client-ldap.properties \ --topic machine-data \ --group machine-data-$RANDOM \ --from-beginning
You see the data produced by the first session continuously appear on the screen as output.
Leave the second session also running.
- Navigate to , and select the environment where your Kafka cluster is running.
- On the Data Hubs tab of your environment, select the Kafka cluster you created.
Click Streams Messaging Manager on the services pane to
open the SMM web UI.
It may take a few minutes for the consumer or producer activity to start showing in the SMM UI. You may have to refresh your page a few times until the data starts to appear.Once it does, you see one active producer on the left-hand side of the page and at least one active consumer listed on the right-hand side of the page.You also see the LAG metrics listed besides each consumer. This metric indicates the number of messages the consumer is behind the latest message produced to the Kafka topic. You can use this to quickly identify slow consumers that are lagging behind producers.
Click either on the producer or on the consumer, and SMM shows their activity,
highlighting all the topics and partitions they are writing to or reading from,
In this page you can identify metrics for the overall Kafka topic activity and partition-level. The In Sync Replica (ISR) set for each partition can be identified on the screen. The ID of the LEADER replica for each partition is shown on the left-hand side, while the FOLLOWER replicas are represented as teal-colored boxes on the right-hand side.
Click on an empty part of the page to clear the consumer or producer selection,
and then click the profile icon for the
machine-datatopic.This opens the Kafka topic details page where you can find all the topic-related metrics and utilization charts.
Click the DATA EXPLORER tab.
In the Data Explorer page you can sample the data that is flowing through the Kafka topic. Select different partitions and adjust the offset sliders to become familiar with those controls.