What's New in Streams Replication Manager

Prefixless replication with the IdentityReplicationPolicy

Full support, including replication monitoring, is introduced for the IdentityReplicationPolicy. Unlike the DefaultReplicationPolicy, this policy does not rename remote (replicated) topics on target clusters. That is, the topics that you replicate will have the same name in both source and target clusters. This replication policy is recommended for deployments where SRM is used to aggregate data from multiple streaming pipelines. Alternatively, this replication policy can also be used if the deployment requires MirrorMaker1 (MM1) compatible replication.

Prefixless replication is enabled in Cloudera Manager with the Enable Prefixless Replication property. This property configures SRM to use the IdentityReplicationPolicy and enables internal topic based remote topic discovery, which is required for replication monitoring.

Limitations:

Replication loop detection is not supported. As a result, you must ensure that topics are not replicated in a loop between your source and target clusters.
The /v2/topic-metrics/{target}/{downstreamTopic}/{metric} endpoint of SRM Service v2 API does not work properly with prefixless replication. Use the /v2/topic-metrics/{source}/{target}/{upstreamTopic}/{metric} endpoint instead.
The replication metric graphs shown on the Topic Details page of the SMM UI do not work with prefixless replication.

For more information, see

Internal topic based remote topic discovery

From now on, SRM uses an internal Kafka topic to keep track of remote (replicated) topics. Previously, SRM relied on the naming conventions (prefixes) used by the DefaultReplicationPolicy to discover and track remote topics.

This feature enables SRM to provide better monitoring insights on replications. Additionally, if the feature is enabled, SRM is capable of providing replication monitoring even if a replication policy different than the DefaultReplicationPolicy is in use. Most notably, this enables replication monitoring when SRM is configured for prefixless replication with the IdentityReplicationPolicy.

This feature is enabled in Cloudera Manager by selecting the Remote Topics Discovery With Internal Topic property. The property is selected by default on newly deployed clusters, but must be enabled manually for existing clusters after an upgrade. Cloudera recommends that you enable this feature no matter what replication policy you are using.

For more information, see Streams Replication Manager remote topic discovery

Improvements related to raw metric collection and aggregation

New metric replication-records-lag

The SRM Service now reports a new metric on its REST API called replication-records-lag. This metric provides information regarding the replication lag based on offsets. The metric is available both on the cluster and the topic level.

Raw metrics are compressed using LZ4

SRM internal metric producers from now on use LZ4 compression by default. LZ4 was chosen as it provides the best combination in terms of compression speed and performance. As a result, Cloudera recommends that you use LZ4. If required, however, you can change the compression by doing the following:

Add the following configuration entries to Streams Replication Manager's Replication Configs:

workers.cloudera.metrics.reporter.producer.compression.type=[***COMPRESSION***]

connectors.cloudera.metrics.reporter.producer.compression.type=[***COMPRESSION***]

Add the following to Additional Configs For Streams Application Running Inside SRM Service:
```
producer.compression.type=[***COMPRESSION***]
```

For more information regarding, metrics, monitoring, as well as raw metric collection and aggregation, see Streams Replication Manager monitoring and metrics .

Improved SRM logging

The logging capabilities of SRM are improved. The following improvements have been introduced:

Kafka clients created by the internal connectors of SRM reference the replication flow the connectors are a part of (KAFKA-14838 backport).
SRM now includes references to the replication flow in the log context of its internal connectors.

These changes enable differentiation between the logs associated with each replication flow.

Metrics and health checks for the status processor Streams application in SRM Service

SRM Service health tests now show the state of the Connect status processor Streams application.

SRM topic creation timeout increased

Streams Replication Manager internal topic creation timeout property defaults are increased to 20s to tolerate intermittent issues at startup.