Automatic group offset synchronization
Automatic group offset synchronization is a feature in Streams Replication Manager (SRM) that automates the export and application of translated consumer group offsets. Enabling this feature can simplify the manual steps that you need to take to migrate consumer groups in a failover or failback scenario.
SRM automatically translates consumer group offsets between clusters. While
the offset mappings are created by SRM, they are not applied by default to the consumer groups of
the target cluster. As a result, by default, migrating consumer groups from one cluster to
another involves running the srm-control offsets
and
kafka-consumer-groups
tools. The srm-control offsets
tool
exports translated offsets, kafka-consumer-groups
resets and applies the
translated offsets on the target cluster.
You can automate this process by enabling automatic group offset synchronization. If automatic
group offset synchronization is enabled, the translated group offsets of the source cluster are
automatically exported from the source cluster and are applied on the target cluster (they are
written to the __consumer_offsets
topic ). If you choose to enable this feature,
running srm-control offsets
and kafka-consumer-groups
is not
required to migrate consumer groups. You only need to restart and redirect consumers to consume
from the new cluster.
- Automatic group offset synchronization does not fully automate a failover or failback process. It only allows you to skip certain manual steps. Consumers must be restarted and redirected to the new cluster even if the feature is enabled.
- Offsets are synced at a configured interval. As a result, it is not guaranteed that the
latest translated offsets are applied. If you want to have the latest offsets applied, Cloudera
recommends that you export and apply consumer group offsets manually. The exact interval
depends on
sync.group.offsets.interval.seconds
andemit.checkpoints.interval.seconds
. - The checkpointing frequency configured for SRM can have an effect on the group offset
synchronization frequency. The frequency of group offset synchronization can be configured with
sync.group.offsets.interval.seconds
. However, specifying an interval using this property might not result in the offsets being synchronized at the set frequency. This depends on howemit.checkpoints.interval
is configured. Theemit.checkpoints.interval
property specifies how frequently offset information is fetched (checkpointing). Because offset synchronization can only happen after offset information is available, the frequency configured with theemit.checkpoints.interval
property might introduce additional latency. For example, assume that you set offset synchronization to 60 seconds (default), but have checkpointing set to 120 seconds. In such a case, offset synchronization happens every 60 seconds, but because offset information is only refreshed every 120 seconds, in practice offsets are only synchronized every 120 seconds. - Offsets are only synchronized for the consumers that are inactive in the target cluster. This is done so that the SRM does not override the offsets in the target cluster.
- During a failover-failback scenario, if a consumer group makes progress in the replica topic in the backup cluster and a failback occurs, the progress made in the replica topic is lost, and messages are re-processed from the original topic.
- By default, SRM checkpointing only uses the latest offset sync record, which means that
consumer groups behind the latest mapping cannot be checkpointed. If the consumer group is
behind by more than the value of
offset.lag.max
property in replicating the records to the offset sync, the offset translation is skipped because the offset is too far in the past to translate accurately. - SRM checkpointing reads the offset-syncs topic to create offset mappings for committed consumer group offsets. In some cases, a mapping might not available in offset syncs. In such a case, SRM copies the source offset, which might not be a valid offset in the replica topic. This can cause issues when auto offset sync is also turned on.
- If
sync.group.offsets.enabled
is set totrue
, Kafka only emits checkpoints for partitions that have offset syncs. Partitions for which no offsets are committed (because they cannot be translated) do not appear when describing consumer offsets in the target cluster.