Introduction to Streams Messaging Manager

Streams Messaging Manager is an operations monitoring and management tool that provides end-to-end visibility in an enterprise Apache Kafka® environment.

With Streams Messaging Manager, you can gain clear insights about your Kafka clusters. You can understand the end-to-end flow of message streams from producers to topics to consumers. Streams Messaging Manager helps you troubleshoot your Kafka environment to identify bottlenecks, throughputs, consumer patterns, traffic flow etc. Streams Messaging Manager enables you to analyze the stream dynamics between producers and consumers using various filters. You can optimize your Kafka environment based on the key performance insights gathered from various brokers and topics. With the tight integration of Apache Atlas, you can gain complete data lineage across multiple Kafka hops, producers and consumers with powerful data flow visualization.

Simplifies troubleshooting Kafka environments

Streams Messaging Manager provides intelligence-based filtering that allows a user to select a producer, broker, topic or consumer and see only related entities based on the selection. Streams Messaging Manager is smart enough to show only those producers that are sending data to the selected topics and show only those consumer groups that are consuming from those topics. The filtering works on the selection of any of the four entities. This enables users to quickly hone in on the root cause when troubleshooting and debugging Kafka issues.

Visualizes end-to-end Kafka stream flows

Another powerful feature of Streams Messaging Manager is its ability to visualize all the data streams/flows across all your Kafka clusters. You can select any entity and visualize how data flows with respect to the entity selected.

Extends monitoring and management capabilities with REST API

Streams Messaging Manager offers REST endpoints for all of its capabilities. This enables developers to integrate Streams Messaging Manager with their other enterprise tools such as APM or case or ticketing systems.

Tracks data lineage and governance from edge-to-enterprise

Streams Messaging Manager has been fully integrated with Apache Atlas for governance and data lineage, Apache Ranger for security and access control management, Apache Ambari for infrastructure level monitoring and lifecycle actions for the cluster, and Grafana to be able to graph Kafka metrics over time.

Visualize lineage between producers and consumers

With Streams Messaging Manager, you can visualize the lineage between producers and consumers.

Lineage information helps you to understand how the message is moving from a producer to a consumer group and which topics or partitions are part of that flow. This is just a high level information, intended for basic monitoring. For the fully featured lineage monitoring, see Atlas and its Kafka monitoring capabilities in Kafka lineage.