Streams Replication Manager requirements

When planning and designing your Streams Replication Manager (SRM) deployment that has multiple installation of SRM, there are a number of practices that you must follow. Otherwise, you might encounter issues when you are replicating data with SRM.

Kafka cluster aliases must be uniform across clusters

When you have multiple SRM installations, the cluster aliases must be defined the same way in each installation’s configuration. SRM utilizes the cluster alias to detect replication cycles and as a result avoid infinite replication loops.

The heartbeats topic must be present on all source clusters

To be able to properly monitor and discover replications, the heartbeats topic must be present on all source clusters. The heartbeats topic is created by the SRM Driver when a target cluster is set with the Streams Replication Manager Driver Target Cluster property and when heartbeating is enabled for the replication. This means that to ensure that the heartbeats topic is created, and has data written into it periodically, at least a single SRM Driver cluster must target each source cluster.

Driver targets must be managed correctly

The Streams Replication Manager Cluster alias and the Streams Replication Manager Driver Target Cluster properties must be managed correctly when you have multiple SRM installations. If a target is specified for the replication in the Streams Replication Manager Driver Target Cluster property, a Connect worker is created for each possible cluster pair based on the aliases present in Streams Replication Manager Cluster alias. As a result of this, it is possible for different SRM Driver clusters to merge through their Connect workers. To avoid this, ensure that different SRM Driver installations target different clusters.

Service targets must be managed correctly

The Streams Replication Manager Cluster alias and the Streams Replication Manager Service Target Cluster properties must be managed correctly when you have multiple SRM installations. If a target is specified for monitoring in the Streams Replication Manager Service Target Cluster property, a Kafka Streams application instance is created for each cluster present in the Streams Replication Manager Service Target Cluster property. As a result of this, it is possible for different SRM Service clusters to merge through their Kafka Streams application instances. To avoid this, ensure that different SRM Service installations target different clusters.