Learn how to configure SRM for failover and failback.
To prepare for a failover or failback scenario you have to set up SRM with bidirectional
replication. Additionally, you have to make sure that all mission critical topics and
consumer groups are whitelisted on both the primary and backup clusters.
-
In Cloudera Manager select Streams Replication Manager.
- Go to Configuration.
-
Set up bidirectional replication between clusters:
- Find the Streams Replication Manager
Cluster alias. property.
- Add a comma
delimited list of cluster aliases. For example:
- Find the
streams.replication.manager's replication configs
property.
- Click the add button and add new lines
for each cluster alias you have specified in the Streams Replication
Manager Cluster alias. property
- Add connection information for your
clusters. For example:
primary.bootstrap.servers=primary_host1:9092,primary_host2:9092,primary_host3:9092
secondary.bootstrap.servers=secondary_host1:9092,secondary_host2:9092,secondary_host3:9092
Each cluster has to be added to a new line. If a cluster has multiple hosts, add
them to the same line but delimit them with commas.
- Click the add button and add new lines
for each unique replication you want to add and enable.
- Add and enable your replications. For example:
primary->secondary.enabled=true
secondary->primary.enabled=true
- Whitelist required consumer groups and topics on the primary cluster.
- Whitelist groups:
srm-control groups --source [PRIMARY_CLUSTER] --target [SECONDARY_CLUSTER] --add [GROUP1],[GROUP2]
- Whitelist topics:
srm-control topics --source [PRIMARY_CLUSTER] --target [SECONDARY_CLUSTER] --add [TOPIC1],[TOPIC2]
- Whitelist required remote topics and consumer groups on the secondary
cluster.
- Whitelist remote groups:
srm-control groups --source [SECONDARY_CLUSTER] --target [PRIMARY_CLUSTER] --add [GROUP1],[GROUP2]
- Whitelist remote topics:
srm-control topics --source [SECONDARY_CLUSTER] --target [PRIMARY_CLUSTER] --add [PRIMARY_CLUSTER.TOPIC1],[PRIMARY_CLUSTER.TOPIC2]
- Verify that all required topics and consumer groups are whitelisted.
- Verify consumer groups:
srm-control groups --source [PRIMARY_CLUSTER] --target [SECONDARY_CLUSTER] --list
srm-control groups --source [SECONDARY_CLUSTER] --target [PRIMARY_CLUSTER] --list
- Verify topics:
srm-control topics --source [PRIMARY_CLUSTER] --target [SECONDARY_CLUSTER] --list
srm-control topics --source [SECONDARY_CLUSTER] --target [PRIMARY_CLUSTER] --list
SRM is set up with bidirectional replication and all mission critical topics and consumer
groups are whitelisted on both the primary and secondary clusters.