Bidirectional Replication of Active Clusters
Configuration example for two active Kafka clusters setup with bidirectional replication.
A typical scenario involves two active Kafka clusters within the same region but in separate availability zones. Clients can connect to either cluster in case one is temporarily unavailable. This example demonstrates the steps required to set up a deployment with two clusters configured with bidirectional replication. Additionally, it also provides example commands to start replication between clusters.
Install the following configuration file on every broker.
# two clusters: primary and secondary clusters = primary, secondary primary.bootstrap.servers = primary_host1:9092, primary_host2:9092, primary_host3:9092 secondary.bootstrap.servers = secondary_host1:9092, secondary_host2:9092, secondary_host3:9092 # bidirectional replication between primary and secondary primary->secondary.enabled = true secondary->primary.enabled = true
- Run the
--clustersoption on every Kafka broker host.Running the
--clustersoption allows you to specify which clusters each instance of the driver should write to. In this example, both instances of the driver are set up to read data from all clusters but only write to the cluster they are running on. This allows you to distribute replication workloads.
srm-driver --clusters primary
srm-driver --clusters secondary
- Optional: If you want to monitor replication, run
srm-serviceon one host per cluster. Each instance of the service should target the cluster its running on.
# on a single primary host srm-service --target primary # on single secondary host srm-service --target secondary
- Replicate data between clusters with the following commands:
srm-control topics --source primary --target secondary --add ".*"
srm-control topics --source secondary --target primary --add ".*"