What's New in Streams Replication Manager
Learn about the new features of Streams Replication Manager in Cloudera Runtime 7.2.15.
SRM now creates all internal topics with correct configurations at startup
The internal topics used by SRM are now automatically created with correct configurations
at startup. These are the metrics topics, the topics used by the
srm-control tool, and the topics used by the SRM Service for service
discovery. Additionally, SRM also verifies that the topics are created with correct
configurations. If the topics are not configured as expected, SRM fails to start. This
improvement fixes CDPD-31745.
Increase the default replication factor of internal topics to 3
The internal topics used by SRM are now created with a replication factor of 3 by default. As a result, SRM is now more resistant to host failures. Additionally, Cruise Control can now automatically heal SRM’s internal topics in the event of a single host failure.
SRM now waits for latest offset syncs and does not set the consumer offset into the future
The MirrorCheckpointConnector now checks the latest message in the offset sync topic at startup, and does not emit a checkpoint message until it has read from the beginning all the messages prior and including that last message.
As a part of this improvement, a new configuration property,
emit.checkpoints.end.offset.protection is introduced. When this property
is enabled, the MirrorCheckpointTask checks the end offset of the replicated topic prior to
emitting a checkpoint, and limits the replicated offset to be maximum that value. With this
behavior enabled, SRM no longer encounters an issue where in certain situations the
replicated offset could be higher than the end offset of the replicated topic, producing a
negative lag. The property is enabled by default, but can be configured using the
Streams Replication Manager's Replication Configs property.