Streams Replication Manager requirements

When planning and designing your Streams Replication Manager (SRM) deployment that has multiple instances of the SRM service, there are a number of practices that you must follow. Otherwise, you might encounter issues when you are replicating data with SRM.

Kafka cluster aliases must be uniform across clusters

When you have multiple SRM services, the cluster aliases must be defined identically in all instances. SRM utilizes the cluster alias to detect replication loops.

The heartbeats topic must be present on all source clusters

To be able to properly monitor and discover replications, the heartbeats topic must be present on all source clusters. The heartbeats topic is created by the SRM Driver when a target cluster is set with the Streams Replication Manager Driver Target Cluster property and when heartbeating is enabled for the replication. This means that to ensure that the heartbeats topic is created, and has data written into it periodically, at least a single SRM Driver cluster must target each source cluster.

Driver targets must be managed correctly

The Streams Replication Manager Cluster alias and the Streams Replication Manager Driver Target Cluster properties must be managed correctly when you have multiple SRM services. If a target is specified for the replication in the Streams Replication Manager Driver Target Cluster property, a Connect worker is created for each possible cluster pair based on the aliases present in Streams Replication Manager Cluster alias. As a result of this, it is possible for different SRM Driver clusters to merge through their Connect workers. To avoid this, ensure that different SRM Driver clusters target different Kafka clusters.

Service targets must be managed correctly

The Streams Replication Manager Cluster alias and the Streams Replication Manager Service Target Cluster properties must be managed correctly when you have multiple SRM services. If a target is specified for monitoring in the Streams Replication Manager Service Target Cluster property, a Kafka Streams application instance is created for each cluster present in the Streams Replication Manager Service Target Cluster property. As a result of this, it is possible for different SRM Service clusters to merge through their Kafka Streams application instances. To avoid this, ensure that different SRM Service clusters target different Kafka clusters.

All SRM services must use the same replication policy

SRM ships with and supports multiple replication policies. Using a mix of these policies in a single deployment is not supported. Decide which replication policy you want to use and ensure that all SRM services are configured to use your chosen policy. In addition transitioning from one policy to another is not recommended.

Ensure that you do not create replication loops if using the IdentityReplicationPolicy

The IdentityReplicationPolicy cannot detect replication loops. As a result, if you are planning to create a setup with bidirectional replication using the IdentityReplicationPolicy, you must ensure that you do not create any replication loops when you set up your allowlists and denylists (topic filters) with srm-control.