Learn how to migrate consumers between clusters.
If a primary Kafka cluster is temporarily unavailable, you can
migrate mission-critical workloads to a secondary Kafka cluster (failover). When the primary
cluster is restored, you can migrate back (failback). The steps for migrating consumers in a
failover or failback scenario are identical. However, depending on the scenario, your source
and target clusters will be different. During failover you migrate consumers from primary to
secondary, while during failback you migrate consumers from secondary to primary.
- Make sure that the clusters that you are migrating consumers between are set up with
bidirectional replication.
- Verify that all mission critical consumer groups and topics, including the ones on
the secondary cluster are whitelisted.
-
Export the translated consumer group offsets of the source cluster:
srm-control offsets --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --group [GROUP1] --export > out.csv
- Reset consumer offsets on the target cluster:
kafka-consumer-groups --bootstrap-server [TARGET_BROKER:PORT] --reset-offsets --group [GROUP1] --execute --from-file out.csv
- Start consumers on the target cluster.
Consumers automatically resume processing messages on the target cluster where they left
off on the source cluster.
