Introduction to Streams Messaging Manager

Streams Messaging Manager (SMM) is an operations monitoring and management tool that provides end-to-end visibility in an enterprise Apache Kafka® environment.

With SMM, you can gain clear insights about your Kafka clusters. You can understand the end-to-end flow of message streams from producers to topics to consumers. SMM helps you troubleshoot your Kafka environment to identify bottlenecks, throughputs, consumer patterns, traffic flow etc. SMM enables you to analyze the stream dynamics between producers and consumers using various filters. You can optimize your Kafka environment based on the key performance insights gathered from various brokers and topics. With the tight integration of Apache Atlas, you can gain complete data lineage across multiple Kafka hops, producers and consumers with powerful data flow visualization.

Simplifies troubleshooting Kafka environments

SMM provides intelligence-based filtering that allows a user to select a producer, broker, topic or consumer and see only related entities based on the selection. SMM is smart enough to show only those producers that are sending data to the selected topics and show only those consumer groups that are consuming from those topics. The filtering works on the selection of any of the four entities. This enables users to quickly hone in on the root cause when troubleshooting and debugging Kafka issues.

Visualizes end-to-end Kafka stream flows

Another powerful feature of SMM is its ability to visualize all the data streams/flows across all your Kafka clusters. You can select any entity and visualize how data flows with respect to the entity selected.

Extends monitoring and management capabilities with REST API

SMM offers REST endpoints for all of its capabilities. This enables developers to integrate SMM with their other enterprise tools such as APM or case or ticketing systems.

Tracks data lineage and governance from edge-to-enterprise

SMM has been fully integrated with Apache Atlas for governance and data lineage, Apache Ranger for security and access control management, Apache Ambari for infrastructure level monitoring and lifecycle actions for the cluster, and Grafana to be able to graph Kafka metrics over time.

Visualize lineage between producers and consumers

With SMM, you can visualize the lineage between producers and consumers.

Lineage information helps you to understand how the message is moving from a producer to a consumer group and which topics or partitions are part of that flow. This is just a high level information, intended for basic monitoring. For the fully featured lineage monitoring, see Atlas and its Kafka monitoring capabilities in Kafka lineage.