Cloudera Docs

Automatic group offset synchronization

Automatic group offset synchronization is a feature in Streams Replication Manager that automates the export and application of translated consumer group offsets. Enabling this feature can simplify the manual steps that you need to take to migrate consumer groups in a failover or failback scenario.

Streams Replication Manager automatically translates consumer group offsets between clusters. While the offset mappings are created by Streams Replication Manager, they are not applied by default to the consumer groups of the target cluster. As a result, by default, migrating consumer groups from one cluster to another involves running the srm-control offsets and kafka-consumer-groups tools. The srm-control offsets tool exports translated offsets, kafka-consumer-groups resets and applies the translated offsets on the target cluster.

You can automate this process by enabling automatic group offset synchronization. If automatic group offset synchronization is enabled, the translated group offsets of the source cluster are automatically exported from the source cluster and are applied on the target cluster (they are written to the __consumer_offsets topic ). If you choose to enable this feature, running srm-control offsets and kafka-consumer-groups is not required to migrate consumer groups. You only need to restart and redirect consumers to consume from the new cluster.

Although automatic group offset synchronization can simplify migrating consumer groups, ensure that you understand the following about its behavior:
  • Automatic group offset synchronization does not fully automate a failover or failback process. It only allows you to skip certain manual steps. Consumers must be restarted and redirected to the new cluster even if the feature is enabled.
  • Offsets are synced at a configured interval. As a result, it is not guaranteed that the latest translated offsets are applied. If you want to have the latest offsets applied, Cloudera recommends that you export and apply consumer group offsets manually. The exact interval depends on sync.group.offsets.interval.seconds and emit.checkpoints.interval.seconds.
  • The checkpointing frequency configured for Streams Replication Manager can have an effect on the group offset synchronization frequency. The frequency of group offset synchronization can be configured with sync.group.offsets.interval.seconds. However, specifying an interval using this property might not result in the offsets being synchronized at the set frequency. This depends on how emit.checkpoints.interval is configured. The emit.checkpoints.interval property specifies how frequently offset information is fetched (checkpointing). Because offset synchronization can only happen after offset information is available, the frequency configured with the emit.checkpoints.interval property might introduce additional latency. For example, assume that you set offset synchronization to 60 seconds (default), but have checkpointing set to 120 seconds. In such a case, offset synchronization happens every 60 seconds, but because offset information is only refreshed every 120 seconds, in practice offsets are only synchronized every 120 seconds.
  • Offsets are only synchronized for the consumers that are inactive in the target cluster. This is done so that the Streams Replication Manager does not override the offsets in the target cluster.
  • During a failover-failback scenario, if a consumer group makes progress in the replica topic in the backup cluster and a failback occurs, the progress made in the replica topic is lost, and messages are re-processed from the original topic.
Additionally, Streams Replication Manager checkpointing behavior might affect automatic group offset synchronization in the following ways:
  • By default, Streams Replication Manager checkpointing only uses the latest offset sync record, which means that consumer groups behind the latest mapping cannot be checkpointed. If the consumer group is behind by more than the value of offset.lag.max property in replicating the records to the offset sync, the offset translation is skipped because the offset is too far in the past to translate accurately.
  • Streams Replication Manager checkpointing reads the offset syncs topic to create offset mappings for committed consumer group offsets. In some cases, a mapping might not available in offset syncs. In such a case, Streams Replication Manager copies the source offset, which might not be a valid offset in the replica topic. This can cause issues when auto offset sync is also turned on.
  • If sync.group.offsets.enabled is set to true, Kafka only emits checkpoints for partitions that have offset syncs. Partitions for which no offsets are committed (because they cannot be translated) do not appear when describing consumer offsets in the target cluster.