Streams Messaging Manager and Apache NiFi

Streams Messaging Manager (SMM) is an operations monitoring and management tool that provides end-to-end visibility in an enterprise Apache Kafka environment. SMM is used with Schema Registry, which provides a shared repository of schemas that allows applications to flexibly interact with each other.

You can use Apache NiFi to develop, manage, and monitor data flows. The Apache NiFi Registry provides a central location for the storage and management of shared data flow resources across more than one instance of Apache NiFi.

SMM and Schema Registry

Monitoring Kafka Clusters

How to use Streams Messaging Manager to monitor Apache Kafka clusters, producers, topics, brokers, and consumers.

Managing Alert Policies

How to use Streams Messaging Manager to create, modify, and monitor alert policies and alerts.

Managing Topics

How to create, moidify, and delete Kafka topics.

Monitoring End to End Latency

Describes how you can use Streams Messaging Manager to monitor end-to-end latency in topics. Use the latency feature to verify whether consumers are consuming the messages produced in a topic within the time SLA, identify the slow consuming applications that might occur due to an external or process bottleneck, and verify whether the consumers are consuming all the messages.

Monitoring Kafka Cluster Replications

Describes how you can use Streams Replication Manager to implement cross-cluster Kafka topic replication in Streams Messaging Manager.

Using Schema Registry

How to use Schema Registry.

Apache NiFi and NiFi Registry

Using Apache NiFi

Apache NiFi is a dataflow system based on the concepts of flow-based programming. It supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic.

Using Apache NiFi Registry

How to use the NiFi Registry to manage DataFlow versions.