Managing topics across multiple Kafka clusters

You may have more than one Kafka cluster to support:

Geographic distribution
Disaster recovery
Organizational requirements

You can distribute messages across multiple clusters. It can be handy to have a copy of one or more topics from other Kafka clusters available to a client on one cluster. Mirror Maker is a tool that comes bundled with Kafka to help automate the process of mirroring or publishing messages from one cluster to another. "Mirroring" occurs between clusters where "replication" distributes message within a cluster.

Figure 1. Mirror Maker Makes Topics Available on Multiple Clusters

While the diagram shows copying to one topic, Mirror Maker’s main mode of operation is running continuously, copying one or more topics from the source cluster to the destination cluster.

Keep in mind the following design notes when configuring Mirror Maker:

Mirror Maker runs as a single process.
Mirror Maker can run with multiple consumers that read from multiple partitions in the source cluster.
Mirror Maker uses a single producer to copy messages to the matching topic in the destination cluster.