Using Streams Replication ManagerPDF version

Migrating Consumer Groups Between Clusters

Learn how to migrate consumers between clusters.

If a primary Kafka cluster is temporarily unavailable, you can migrate mission-critical workloads to a secondary Kafka cluster (failover). When the primary cluster is restored, you can migrate back (failback). The steps for migrating consumers in a failover or failback scenario are identical. However, depending on the scenario, your source and target clusters will be different. During failover you migrate consumers from primary to secondary, while during failback you migrate consumers from secondary to primary.
  • Make sure that the clusters that you are migrating consumers between are set up with bidirectional replication.
  • Verify that all mission critical consumer groups and topics, including the ones on the secondary cluster are whitelisted.
  • The steps you need to take differ depending on whether automatic group offset synchronization is enabled. Check SRM's configuration to see if the feature is enabled. For more information, see Configuring automatic group offset synchronization
  1. Migrate consumer group offsets.
    If automatic group offset synchronization is disabled follow the steps provided under Manual migration. If automatic group offset synchronization is enabled, follow the steps under Automatic migration.
    1. Export the translated consumer group offsets of the source cluster.
      srm-control offsets --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --group [GROUP1] --export > out.csv
    2. Reset consumer offsets on the target cluster.
      kafka-consumer-groups --bootstrap-server [TARGET_BROKER:PORT] --reset-offsets --group [GROUP1] --execute --from-file out.csv
  2. Start consumers on the target cluster.

Consumers automatically resume processing messages on the target cluster where they left off on the source cluster.