Mirroring data between Kafka clusters
Also available as:

Running MirrorMaker on Kerberos-Enabled Clusters

To run MirrorMaker on a Kerberos/SASL-enabled cluster, configure producer and consumer properties as follows:

  1. Choose or add a new principal for MirrorMaker. Do not use kafka or any other service accounts. The following example uses principal mirrormaker.
  2. Create client-side Kerberos keytabs for your MirrorMaker principal. For example:
    sudo kadmin.local -q "ktadd -k /tmp/mirrormaker.keytab mirrormaker/HOSTNAME@EXAMPLE.COM"
  3. Add a new Jaas configuration file to the node where you plan to run MirrorMaker:
  4. Add the following settings to the KafkaClient section of the new Jaas configuration file. Make sure the principal has permissions on both the source cluster and the target cluster.
    KafkaClient {
         com.sun.security.auth.module.Krb5LoginModule required
  5. Run the following ACL command on the source and destination Kafka clusters:
    --topic test-topic 
    --allow-principal user:mirrormaker 
    --operation ALL 
    --config /usr/hdp/current/kafka-broker/config/server.properties
  6. In your MirrorMaker consumer.config and producer.config files, specify security.protocol=SASL_PLAINTEXT.
  7. Start MirrorMaker. Specify the new.consumer option in addition to your other options. For example:
    /usr/hdp/current/kafka-broker/bin/kafka-run-class.sh kafka.tools.MirrorMaker 
    --consumer.config consumer.properties 
    --producer.config target-cluster-producer.properties 
    --whitelist my-topic