Monitoring Kafka

Learn how to monitor Kafka clusters, producers, consumers topics brokers, log size and lineage using Streams Messaging Manager (SMM).

Monitoring Kafka clusters
The overview page provides you with tools to see a snapshot of the Kafka cluster you are monitoring. After you select the Kafka cluster to monitor, you can see the total number of producers, brokers, topics, and consumer groups in that cluster. You can also monitor producer and consumer metrics.
Monitoring Kafka producers
By monitoring Kafka producers, you can track the active and inactive producers in your cluster. You can also change the period of time after which a producer is considered inactive.
Monitoring Kafka topics
By monitoring Kafka topics, you can track the total number of topics in your cluster and details about the topics. You can also monitor Grafana metrics for the topics in your cluster.
Monitoring Kafka brokers
By monitoring Kafka brokers, you can track various details about brokers including the host where the broker is located, disk space used by the broker, throughput, messages coming in, partitions, and replicas.
Monitoring Kafka consumers
By monitoring Kafka consumer groups, you can track active and passive consumer groups, or all consumer groups, which use the default internal __consumer_offsets topic to store the consumed offset information. You can track additional details about consumer groups. You can also track details including number of consumers and consumer instances included in a group and consumer group lag in the consumer group profile.
Monitoring log size information
The SMM UI shows log size related information about brokers, topics, and partitions. Furthermore, warning messages appear when log directory related errors happen.
Monitoring lineage information
Learn how you can visualize the lineage between producers and consumers.