Configuration example for two active Kafka clusters setup with bidirectional
replication.
A typical scenario involves two active Kafka clusters within the same region but in
separate availability zones. Clients can connect to either cluster in case one is
temporarily unavailable. This example demonstrates the steps required to set up a deployment
with two clusters configured with bidirectional replication. Additionally, it also provides
example commands to start replication between clusters.
Figure 1. Bidirectional Replication of Active
Clusters
Install the following configuration file on every broker.
# two clusters: primary and secondary
clusters = primary, secondary
primary.bootstrap.servers = primary_host1:9092, primary_host2:9092, primary_host3:9092
secondary.bootstrap.servers = secondary_host1:9092, secondary_host2:9092, secondary_host3:9092
# bidirectional replication between primary and secondary
primary->secondary.enabled = true
secondary->primary.enabled = true
Run the srm-driver with the
--clusters option on every Kafka broker host.
Running the
srm-driver with the --clusters option allows you to
specify which clusters each instance of the driver should write to. In this example, both
instances of the driver are set up to read data from all clusters but only write to the
cluster they are running on. This allows you to distribute replication workloads.
On primary
hosts:
srm-driver --clusters primary
On
secondaryhosts:
srm-driver --clusters secondary
Optional: If you want to monitor replication, run
srm-service on one host per cluster. Each instance of the service
should target the cluster its running on.
# on a single primary host
srm-service --target primary
# on single secondary host
srm-service --target secondary
Replicate data between clusters with the following commands:
