Automatic group offset synchronization
Automatic group offset synchronization is a feature in Streams Replication Manager that automates the export and application of translated consumer group offsets. Enabling this feature can simplify the manual steps that you need to take to migrate consumer groups in a failover or failback scenario.
Streams Replication Manager automatically translates consumer group offsets between
clusters. While the offset mappings are created by Streams Replication Manager, they are
not applied by default to the consumer groups of the target cluster. As a result, by default,
migrating consumer groups from one cluster to another involves running the srm-control
offsets and kafka-consumer-groups tools. The srm-control
offsets tool exports translated offsets, kafka-consumer-groups resets
and applies the translated offsets on the target cluster.
You can automate this process by enabling automatic group offset synchronization. If automatic
group offset synchronization is enabled, the translated group offsets of the source cluster are
automatically exported from the source cluster and are applied on the target cluster (they are
written to the __consumer_offsets topic ). If you choose to enable this feature,
running srm-control offsets and kafka-consumer-groups is not
required to migrate consumer groups. You only need to restart and redirect consumers to consume
from the new cluster.
- Automatic group offset synchronization does not fully automate a failover or failback process. It only allows you to skip certain manual steps. Consumers must be restarted and redirected to the new cluster even if the feature is enabled.
- Offsets are synced at a configured interval. As a result, it is not guaranteed that the
latest translated offsets are applied. If you want to have the latest offsets applied, Cloudera
recommends that you export and apply consumer group offsets manually. The exact interval
depends on
sync.group.offsets.interval.secondsandemit.checkpoints.interval.seconds. - The checkpointing frequency configured for Streams Replication Manager can have an
effect on the group offset synchronization frequency. The frequency of group offset
synchronization can be configured with
sync.group.offsets.interval.seconds. However, specifying an interval using this property might not result in the offsets being synchronized at the set frequency. This depends on howemit.checkpoints.intervalis configured. Theemit.checkpoints.intervalproperty specifies how frequently offset information is fetched (checkpointing). Because offset synchronization can only happen after offset information is available, the frequency configured with theemit.checkpoints.intervalproperty might introduce additional latency. For example, assume that you set offset synchronization to 60 seconds (default), but have checkpointing set to 120 seconds. In such a case, offset synchronization happens every 60 seconds, but because offset information is only refreshed every 120 seconds, in practice offsets are only synchronized every 120 seconds. - Offsets are only synchronized for the consumers that are inactive in the target cluster. This is done so that the Streams Replication Manager does not override the offsets in the target cluster.
- During a failover-failback scenario, if a consumer group makes progress in the replica topic in the backup cluster and a failback occurs, the progress made in the replica topic is lost, and messages are re-processed from the original topic.
- By default, Streams Replication Manager checkpointing only uses the latest offset
sync record, which means that consumer groups behind the latest mapping cannot be checkpointed.
If the consumer group is behind by more than the value of
offset.lag.maxproperty in replicating the records to the offset sync, the offset translation is skipped because the offset is too far in the past to translate accurately. - Streams Replication Manager checkpointing reads the offset syncs topic to create offset mappings for committed consumer group offsets. In some cases, a mapping might not available in offset syncs. In such a case, Streams Replication Manager copies the source offset, which might not be a valid offset in the replica topic. This can cause issues when auto offset sync is also turned on.
- If
sync.group.offsets.enabledis set totrue, Kafka only emits checkpoints for partitions that have offset syncs. Partitions for which no offsets are committed (because they cannot be translated) do not appear when describing consumer offsets in the target cluster.
