Cloudera Docs

Streams Replication Manager remote topic discovery

Learn how Streams Replication Manager keeps track of remote topics and how this behavior allows Streams Replication Manager to provide full support for prefixless replication with the IdentityreplicationPolicy.

In order to provide replication monitoring, Streams Replication Manager must discover and keep track of the topics that are replicated. By default, Streams Replication Manager achieves this using an internal Kafka topic, which contains information regarding the currently replicated topics. The behavior is as follows:
  1. Streams Replication Manager Drivers periodically check which topics must be replicated.
  2. This check results in a list of source – target topic pairs that must be replicated.
  3. The Streams Replication Manager Driver writes these source – target pairs into the srm-meta.internal Kafka topic, which is an internal compact topic located in the target cluster.
  4. The Streams Replication Manager Service scans srm-meta.internal and consumes the latest message. The latest message contains the currently replicated topics as well as source – target topic name mappings.

Remote topic detection takes into consideration the current state of the replication flow. For example, assume that you have a remote topic, us-east.orders that was replicated at some point in time, but is not currently replicated anymore. Streams Replication Manager correctly identifies that this topic is no longer replicated and will not keep track of it as a remote topic indefinitely. The replication metrics provided by the Streams Replication Manager Service also correctly reflect this state of the topic.

In addition, the use of an internal topic to store replication metadata allows Streams Replication Manager to correctly identify which target maps to which source without relying solely on topic prefixes. This makes it possible for Streams Replication Manager to provide replication monitoring even if a replication policy different than the DefaultReplicationPolicy is in use. Most notably, this enables replication monitoring when Streams Replication Manager is configured for prefixless replication with the IdentityReplicationPolicy.