Migrating Consumer Groups Between Clusters

Learn how to migrate consumers between clusters.

If a primary Kafka cluster is temporarily unavailable, you can migrate mission-critical workloads to a secondary Kafka cluster (failover). When the primary cluster is restored, you can migrate back (failback). The steps for migrating consumers in a failover or failback scenario are identical. However, depending on the scenario, your source and target clusters will be different. During failover you migrate consumers from primary to secondary, while during failback you migrate consumers from secondary to primary.
  • Make sure that the clusters that you are migrating consumers between are set up with bidirectional replication.
  • Verify that all mission critical consumer groups and topics, including the ones on the secondary cluster are whitelisted.
  1. Export the translated consumer group offsets of the source cluster:
    srm-control offsets --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --group [GROUP1] --export > out.csv
  2. Reset consumer offsets on the target cluster:
    kafka-consumer-groups.sh --bootstrap-server [TARGET_BROKER:PORT] --reset-offsets --group [GROUP1] --execute --from-file out.csv
    
  3. Start consumers on the target cluster.

Consumers automatically resume processing messages on the target cluster where they left off on the source cluster.