Monitoring Kafka activity in Streams Messaging Manager

You can use Streams Messaging Manager (SMM) to monitor Kafka activity. To do this, you run two SSH sessions, one for producing data to a Kafka topic and the other to consume the data from the Kafka topic. Then you go to the SMM UI from the Management Console to track the producer and consumer activity.

Now that you know how to configure a Kafka client to connect to the Kafka cluster in CDP, generate some activity and monitor it using SMM. To do this, you collect machine usage data from the broker host using the vmstat command and stream that data into the machine-data topic.

For detailed and comprehensive information about SMM, see Streams Messaging Manager.

Start by opening two SSH connections to the same broker host.
On the first session, run the following command.
```
BROKER_HOST=$(hostname -f)

vmstat 1 1000 | kafka-console-producer \
  --broker-list $BROKER_HOST:9093 \
  --producer.config client-ldap.properties \
  --topic machine-data
```
This command runs vmstat and sends each line from its output as one message every second to the machine-data topic.

This example assumes that you are using LDAP authentication but you can use the authentication method of your preference.

Leave the session running.
On the second SSH session, run the following command to consume the data:
```
BROKER_HOST=$(hostname -f)

kafka-console-consumer \
  --bootstrap-server $BROKER_HOST:9093 \
  --consumer.config client-ldap.properties \
  --topic machine-data \
  --group machine-data-$RANDOM \
  --from-beginning
```
You see the data produced by the first session continuously appear on the screen as output.

Leave the second session also running.
Navigate to Management Console > Environments, and select the environment where your Kafka cluster is running.
On the Data Hubs tab of your environment, select the Kafka cluster you created.
Click Streams Messaging Manager on the services pane to open the SMM web UI.
It may take a few minutes for the consumer or producer activity to start showing in the SMM UI. You may have to refresh your page a few times until the data starts to appear.

Once it does, you see one active producer on the left-hand side of the page and at least one active consumer listed on the right-hand side of the page.

note
SMM shows consumers and producers with some activity in the last few minutes, as active. Because of this, you may see those producers and consumers that finished recently, as active.

You also see the LAG metrics listed besides each consumer. This metric indicates the number of messages the consumer is behind the latest message produced to the Kafka topic. You can use this to quickly identify slow consumers that are lagging behind producers.
Click either on the producer or on the consumer, and SMM shows their activity, highlighting all the topics and partitions they are writing to or reading from, respectively.

In this page you can identify metrics for the overall Kafka topic activity and partition-level. The In Sync Replica (ISR) set for each partition can be identified on the screen. The ID of the LEADER replica for each partition is shown on the left-hand side, while the FOLLOWER replicas are represented as teal-colored boxes on the right-hand side.
Click on an empty part of the page to clear the consumer or producer selection, and then click the profile icon for the machine-data topic.
This opens the Kafka topic details page where you can find all the topic-related metrics and utilization charts.
Click the DATA EXPLORER tab.
In the Data Explorer page you can sample the data that is flowing through the Kafka topic. Select different partitions and adjust the offset sliders to become familiar with those controls.