Streams Replication Manager Service

The Streams Replication Manager Service is responsible for processing the metrics produced by the Streams Replication Manager Drivers. Additionally, it provides a queryable REST API that you can use to monitor and track replications.

Streams Replication Manager Driver roles (SRM Driver) produce raw metrics into the target Kafka clusters. Each replication has a separate raw metric topic. Streams Replication Manager Service roles (SRM Service) run a Kafka Streams application internally, which aggregates the raw metrics. The Streams application ensures that the metrics processing work is load balanced between the SRM Service instances of the same cluster, and that the members of the cluster can coordinate when serving REST API queries.

As a result of this architecture, the SRM Service can only report on replication flows targeting the Kafka cluster which is targeted by the SRM Service.

Multi cluster targeting

The SRM Service is capable of targeting multiple Kafka clusters. In this case, an SRM Service instance spins up a Kafka Streams application for each target Kafka cluster. This way, the SRM Service is able to report on all replication flows targeting any of the configured target clusters.

Remote Querying [Technical Preview]

An SRM Service cluster can be configured to target and gather metrics from multiple clusters (multi cluster targeting). However, a setup like this can result in heavily loaded Service roles, which might not be suitable for your deployment. To circumvent this issue, the SRM Service is capable of connecting to and communicating with other, remote SRM Services to fetch their metrics. This is called Remote Querying. Using Remote Querying makes it possible to designate an SRM Service cluster in your deployment to act as a monitoring gateway. The designated SRM Service cluster then can be used to monitor all clusters and replications in your deployment. For more information, see Remote Querying.

Using the SRM Service REST API

The SRM Service offers a Swagger UI for exploring and querying the REST API. For a user-friendly UI solution, Streams Messaging Manager (SMM) can be integrated with SRM. When SMM is configured to connect to SRM, the Replications page becomes available on the SMM UI. This page displays the information available on the SRM Service REST API. For more information on how you can integrate the two services see Configuring SMM for monitoring Kafka cluster replications.