Set up MirrorMaker in Cloudera Manager
Learn how to set up MirrorMaker in Cloudera Manager
- Consumer/Producer Compatibility
The Mirror Maker consumer needs to be client compatible with the source cluster. The Mirror Maker producer needs to be client compatible with the destination cluster. See Client and Broker Compatibility Across Kafka Versions for further details on compatibility.
- Topic Differences between Clusters
Because messages are copied from the source cluster to the destination cluster—potentially through many consumers funneling into a single producer—there is no guarantee of having identical offsets or timestamps between the two clusters. In addition, as these copies occur over the network, there can be some mismatching due to retries or dropped messages.
- Optimize Mirror Maker Producer Location
- Because Mirror Maker uses a single producer and since producers typically have more difficulty with high latency and/or unreliable connections, it is better to have the producer run “closer” to the destination cluster, meaning in the same data center or on the same rack.
- Destination Cluster Configuration
- Before starting Mirror Maker, make sure that the destination cluster is configured
- Make sure there is sufficient disk space to copy the topic from the source cluster to the destination cluster.
- Make sure the topic exists in the destination cluster or enable topic auto
creation by setting the
auto.create.topics.enableproperty to true. The latter can be done in Cloudera Manager by going to Configuration.
- Kerberos and Mirror Maker
- As mentioned earlier, Mirror Maker runs as a single process. The resulting consumers and producers rely on a single configuration setup. Mirror Maker requires that the source cluster and the destination cluster belong to the same Kerberos realm.
- In Cloudera Manager, select the Kafka service.
- Choose .
- Under Kafka Mirror Maker, click Select hosts.
- Select the host where Mirror Maker will run and click Continue.
- Fill in the Destination Broker List and Source
Broker List with your source and destination Kafka clusters. Use host name, IP address, or fully qualified domain name.
- Fill out the Topic Whitelist. The allowlist (whitelist) is required.
- Fill out the TLS/SSL sections if security needs to be enabled.
- Start the Mirror Maker instance.