Managing Topics across Multiple Kafka Clusters

The Mirror Maker consumer needs to be client compatible with the source cluster. The Mirror Maker producer needs to be client compatible with the destination cluster.

See Client/Broker Compatibility Across Kafka Versions for more details about what it means to be "compatible."

Because messages are copied from the source cluster to the destination cluster—potentially through many consumers funneling into a single producer—there is no guarantee of having identical offsets or timestamps between the two clusters. In addition, as these copies occur over the network, there can be some mismatching due to retries or dropped messages.

Because Mirror Maker uses a single producer and since producers typically have more difficulty with high latency and/or unreliable connections, it is better to have the producer run “closer” to the destination cluster, meaning in the same data center or on the same rack.

As mentioned earlier, Mirror Maker runs as a single process. The resulting consumers and producers rely on a single configuration setup. Mirror Maker requires that the source cluster and the destination cluster belong to the same Kerberos realm.

Where Cloudera Manager is managing the destination cluster:

1. In Cloudera Manager, select the Kafka service.
2. Choose Action > Add Role Instances.
3. Under Kafka Mirror Maker, click Select hosts.
4. Select the host where Mirror Maker will run and click Continue.
5. Fill in the Destination Broker List and Source Broker List with your source and destination Kafka clusters.

   Use host name, IP address, or fully qualified domain name.

6. Fill out the Topic Whitelist.

   The whitelist is required.

7. Fill out the TLS/SSL sections if security needs to be enabled.
8. Start the Mirror Maker instance.