Streams Replication Manager Driver

The Streams Replication Manager Driver is responsible for executing the configured replication work between Kafka clusters. This includes data replication, consumer group offset replication, and heartbeating.

The Streams Replication Manager Driver role is built on top of the Kafka Connect framework and utilizes a group of connectors to execute replication. While Kafka Connect is bound to a single Kafka cluster by design, the Streams Replication Manager Driver must connect to multiple Kafka clusters. The Streams Replication Manager Driver achieves this by wrapping multiple Connect workers in a single driver process. Specifically, for each possible replication between the configured clusters, the Streams Replication Manager Driver spins up a separate Connect worker. The Connect workers join a Connect group which is dedicated to a single replication in the target Kafka cluster. Afterward, each Connect group creates a single MirrorSourceConnector, a single MirrorCheckpointConnector and a single MirrorHeartbeatConnector. These Connectors and the task instances they generate are responsible for different aspects of the replication work.

When Streams Replication Manager Driver High Availability is in use, the Connect workers that are tied to a specific replication, but are running in different Streams Replication Manager Driver processes, coordinate through the Connect group protocol and balance the load.