Streams Replication Manager remote topic discovery

Learn how Streams Replication Manager (SRM) keeps track of remote topics and how this behavior allows SRM to provide full support for prefixless replication with the IdentityreplicationPolicy.

In order to provide replication monitoring, SRM must discover and keep track of the topics that are replicated. By default, SRM achieves this using an internal Kafka topic, which contains information regarding the currently replicated topics. The behavior is as follows:

SRM Drivers periodically check which topics must be replicated.
This check results in a list of source – target topic pairs that must be replicated.
The SRM Driver writes these source – target pairs into the srm-meta.internal Kafka topic, which is an internal compact topic located in the target cluster.
The SRM Service scans srm-meta.internal and consumes the latest message. The latest message contains the currently replicated topics as well as source – target topic name mappings.

Remote topic detection takes into consideration the current state of the replication flow. For example, assume that you have a remote topic, us-east.orders that was replicated at some point in time, but is not currently replicated anymore. SRM correctly identifies that this topic is no longer replicated and will not keep track of it as a remote topic indefinitely. The replication metrics provided by the SRM Service also correctly reflect this state of the topic.

In addition, the use of an internal topic to store replication metadata allows SRM to correctly identify which target maps to which source without relying solely on topic prefixes. This makes it possible for SRM to provide replication monitoring even if a replication policy different than the DefaultReplicationPolicy is in use. Most notably, this enables replication monitoring when SRM is configured for prefixless replication with the IdentityReplicationPolicy.