Streams Replication Manager remote topic discovery
Learn how Streams Replication Manager (SRM) keeps track of remote topics and how this behavior allows SRM to provide full support for prefixless replication with the IdentityreplicationPolicy.
- SRM Drivers periodically check which topics must be replicated.
- This check results in a list of source – target topic pairs that must be replicated.
- The SRM Driver writes these source – target pairs into the
srm-meta.internal
Kafka topic, which is an internal compact topic located in the target cluster. - The SRM Service scans
srm-meta.internal
and consumes the latest message. The latest message contains the currently replicated topics as well as source – target topic name mappings.
Remote topic detection takes into consideration the current state of the replication flow. For
example, assume that you have a remote topic, us-east.orders
that was replicated
at some point in time, but is not currently replicated anymore. SRM correctly identifies that
this topic is no longer replicated and will not keep track of it as a remote topic indefinitely.
The replication metrics provided by the SRM Service also correctly reflect this state of the
topic.
In addition, the use of an internal topic to store replication metadata allows SRM to correctly identify which target maps to which source without relying solely on topic prefixes. This makes it possible for SRM to provide replication monitoring even if a replication policy different than the DefaultReplicationPolicy is in use. Most notably, this enables replication monitoring when SRM is configured for prefixless replication with the IdentityReplicationPolicy.