Migrating Consumer Groups Between Clusters
Learn how to migrate consumers between clusters.
- Make sure that the clusters that you are migrating consumers between are set up with bidirectional replication.
- Verify that all mission critical consumer groups and topics, including the ones on the secondary cluster are whitelisted.
- The steps you need to take differ depending on whether automatic group offset synchronization is enabled. Check SRM's configuration to see if the feature is enabled. For more information, see Configuring automatic group offset synchronization
- Migrate consumer group offsets.If automatic group offset synchronization is disabled follow the steps provided under Manual migration. If automatic group offset synchronization is enabled, follow the steps under Automatic migration.
If automatic group offset synchronization is enabled, the translated offsets of the source cluster are already applied on the target cluster. Continue with Step 2.
- Export the translated consumer group offsets of the source
srm-control offsets --source [SOURCE_CLUSTER] --target [TARGET_CLUSTER] --group [GROUP1] --export > out.csv
- Reset consumer offsets on the target
kafka-consumer-groups --bootstrap-server [TARGET_BROKER:PORT] --reset-offsets --group [GROUP1] --execute --from-file out.csv
- Export the translated consumer group offsets of the source cluster.
- Start consumers on the target cluster.
Consumers automatically resume processing messages on the target cluster where they left off on the source cluster.