Key Features
SRM has the following main features.
Remote topics
SRM replicates Kafka topics from source to target clusters. Remote topics are the replica topics located in target clusters. Remote topics and their source topics are tracked internally by SRM. Additionally, remote topics are by default prefixed with the name (alias) of the source cluster. The naming convention for remote topics is configurable and is determined by the replication policy that is currently in use. For more information, see Replication flows and replication policies.
Consistent semantics
Partitioning and record offsets are synchronized between replicated clusters to ensure consumers can migrate from one cluster to another without losing data or skipping records.
Cross cluster configuration
Topic-level configuration properties are synced across clusters. For example, the cleanup
policy (cleanup.policy), or the log segment file size
(segment.bytes), as well as other topic-level configurations are
automatically synched to remote topics. This simplifies managing topics across multiple
Kafka clusters.
Consumer group checkpoints
In addition to data and configuration, SRM replicates consumer group progress via periodic
checkpoints. At configurable intervals, checkpoint records are emitted to downstream
clusters, encoding the latest offsets for whitelisted consumer groups and topic-partitions.
As with topics, groups are matched against an allowlist which can be updated dynamically with
srm-control. Normally, consumer group offsets are not portable between
Kafka clusters, as offsets are not consistent between otherwise identical topic-partitions
on different clusters. SRM’s checkpoint records account for this by including offsets which
are automatically translated from one cluster to another. This offset translation feature
works in both directions; a consumer group can be migrated from one cluster to another
(failover) and then back again (failback) without skipping records or losing progress.
Automatic topic and partition detection
SRM monitors Kafka clusters for new topics, partitions, and consumer groups as they are created. These are compared with configurable whitelists, which may include regular expressions.
Tooling to automate consumer migration
SRM tooling enables operators to translate offsets between clusters and to migrate consumer groups while preserving state.
Centralized configuration for multi-cluster environments
SRM leverages a single top-level configuration file to enable replication across multiple Kafka clusters. Moreover, command-line tooling can alter which topics and consumer groups are replicated in real-time.
Replication monitoring
Since cluster replication will mainly be used for highly critical Kafka applications, it is crucial for customers to be able to easily and reliably monitor the Kafka cluster replications. The SRM Service collects and aggregates Kafka replication metrics and make them available through a REST API. This REST API is used by Streams Messaging Manager (SMM) to display metrics. Customers could also use the REST API to implement their own monitoring solution or plug it into third party solutions. The metrics make the state of cluster replication visible to end users who then can take corrective action if needed.
Replication policies
The replication policy used by SRM defines the basic rules of how SRM replicates data. SRM ships with two replication policies that are designed for different use cases. These are the DefaultReplicationPolicy, which uses topic name prefixes to provide replication loop detection, and the IdentityReplicationPolicy, which mimics the behavior of MirrorMaker 1 and provides prefixless replication. Both policies also support the monitoring features provided by the SRM Service.
Remote topic discovery and tracking
SRM uses an internal Kafka topic to track which topics are being replicated in a replication flow. This enables the SRM Service to filter topics from monitoring if they are no longer being replicated. Additionally, this feature enables replication monitoring with the SRM Service even if prefixless replication is being used.
