Connectors
Learn about the different Connector implementations created by the Connect workers of the Streams Replication Manager Driver.
- MirrorSourceConnector
- MirrorHeartbeatConnector
- MirrorCheckpointConnector
- MirrorSourceConnector
-
The MirrorSourceConnector is responsible for replicating topics between the source and the target cluster. The topics to be replicated are defined by allow and deny lists, which can be manipulated using the
srm-control
tool.In addition to replicating data, the MirrorSourceConnector also manages the offset sync topic in the source cluster. When production into the target cluster is successful, the mapping between the source offset and the target offset is written into an offset sync topic in the source cluster. The offset sync topic is used by the MirrorCheckpointConnector.
- MirrorHeartbeatConnector
-
The MirrorHeartbeatConnector is responsible for creating the heartbeats topic in the target cluster. It also periodically produces heartbeats into the heartbeats topic. The purpose of the heartbeats topic is twofold:
- It ensures that a topic is available at all times on the source clusters. This way, a replication always has at least a single topic that it can pick up. This functions as a reliable smoke test for the replication.
- It is used by the SRM Service to discover configured replications.
- MirrorCheckpointConnector
-
The MirrorCheckpointConnector is responsible for replicating the committed group offsets between the source and target clusters. The offsets are calculated based on the offset sync topic managed by the MirrorSourceConnector. The offsets of the groups and the topic partitions that are replicated are defined by allow and deny lists, which can be manipulated using the
srm-control
tool.The offsets are written into the checkpoint topic in the target cluster. The contents of the checkpoint topic can be exported with
srm-control
. Exported offsets can then be applied to the consumer groups in the target cluster with thekafka-consumer-groups
Kafka tool.In addition to writing the mapped offsets into the checkpoint topic, the connector is also capable of periodically applying the offsets to the consumer groups in the target cluster. This is done using the Kafka Admin API.