Cloudera Docs

Set up MirrorMaker in Cloudera Manager

Learn how to set up MirrorMaker in Cloudera Manager

Consider the following before setting up MirrorMaker for Kafka:
Consumer/Producer Compatibility

The Mirror Maker consumer needs to be client compatible with the source cluster. The Mirror Maker producer needs to be client compatible with the destination cluster. See Client and Broker Compatibility Across Kafka Versions for further details on compatibility.

Topic Differences between Clusters

Because messages are copied from the source cluster to the destination cluster—potentially through many consumers funneling into a single producer—there is no guarantee of having identical offsets or timestamps between the two clusters. In addition, as these copies occur over the network, there can be some mismatching due to retries or dropped messages.

Optimize Mirror Maker Producer Location
Because Mirror Maker uses a single producer and since producers typically have more difficulty with high latency and/or unreliable connections, it is better to have the producer run “closer” to the destination cluster, meaning in the same data center or on the same rack.
Destination Cluster Configuration
Before starting Mirror Maker, make sure that the destination cluster is configured correctly:
  • Make sure there is sufficient disk space to copy the topic from the source cluster to the destination cluster.
  • Make sure the topic exists in the destination cluster or enable topic auto creation by setting the auto.create.topics.enable property to true. The latter can be done in Cloudera Manager by going to KafkaConfiguration.
Kerberos and Mirror Maker
As mentioned earlier, Mirror Maker runs as a single process. The resulting consumers and producers rely on a single configuration setup. Mirror Maker requires that the source cluster and the destination cluster belong to the same Kerberos realm.
  1. In Cloudera Manager, select the Kafka service.
  2. Choose Action > Add Role Instances.
  3. Under Kafka Mirror Maker, click Select hosts.
  4. Select the host where Mirror Maker will run and click Continue.
  5. Fill in the Destination Broker List and Source Broker List with your source and destination Kafka clusters.
    Use host name, IP address, or fully qualified domain name.
  6. Fill out the Topic Whitelist.
    The allowlist (whitelist) is required.
  7. Fill out the TLS/SSL sections if security needs to be enabled.
  8. Start the Mirror Maker instance.
MirrorMaker is configured and started.