Bidirectional Replication of Active Clusters

Configuration example for two active Kafka clusters setup with bidirectional replication.

A typical scenario involves two active Kafka clusters within the same region but in separate availability zones. Clients can connect to either cluster in case one is temporarily unavailable. This example demonstrates the steps required to set up a deployment with two clusters configured with bidirectional replication. Additionally, it also provides example commands to start replication between clusters.

Figure 1. Bidirectional Replication of Active Clusters

  1. Install the following configuration file on every broker.
    # two clusters: primary and secondary
    clusters = primary, secondary
    primary.bootstrap.servers = primary_host1:9092, primary_host2:9092, primary_host3:9092
    secondary.bootstrap.servers = secondary_host1:9092, secondary_host2:9092, secondary_host3:9092
    # bidirectional replication between primary and secondary
    primary->secondary.enabled = true
    secondary->primary.enabled = true
  2. Run the srm-driver with the --clusters option on every Kafka broker host.
    Running the srm-driver with the --clusters option allows you to specify which clusters each instance of the driver should write to. In this example, both instances of the driver are set up to read data from all clusters but only write to the cluster they are running on. This allows you to distribute replication workloads.
    • On primary hosts:
      srm-driver --clusters primary
    • On secondaryhosts:
      srm-driver --clusters secondary
  3. Optional: If you want to monitor replication, run srm-service on one host per cluster. Each instance of the service should target the cluster its running on.
    # on a single primary host
    srm-service --target primary
    # on single secondary host
    srm-service --target secondary
  4. Replicate data between clusters with the following commands:
    srm-control topics --source primary --target secondary --add ".*"
    srm-control topics --source secondary --target primary --add ".*"