Replicating Data

A step by step guide on how to start replicating data between Kafka clusters with Streams Replication Manager.

Installing and starting Streams Replication Manager on your cluster does not automatically start data replication. In order to start the replication process, you need to update the allowlists with the srm-control tool.

In Cloudera Manager, verify that the Streams Replication Manager driver role is started and is in good health.

Verify that Streams Replication Manager is configured correctly. Make sure that connection information for each Kafka cluster is added as well as at least one source->target replication is specified and enabled.

  1. Update the topics allowlist to start data replication.
    srm-control topics --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --add [TOPIC1],[TOPIC2]
    
  2. Verify that the topics have been added to the allowlist.
    srm-control topics --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --list

The topics you specify with the --add option are added to the topic allowlist and are replicated to the specified target cluster.