What's New in Streams Replication Manager

The --to option in srm-control now creates the file if it does not exist

From now on, srm-control creates the file specified with the --to option if the file does not exist.

Verification of internal metrics topics can now be disabled

A new property, Verify Partition Count Of The Metrics Topic is introduced for the SRM service. This property controls whether SRM verifies the partition count of the srm-metrics-[***SOURCE CLUSTER ALIAS***].internal topics (raw metric topics) when SRM is started. This property is selected by default. Cloudera recommends that you keep this property selected. During certain upgrades, the property is set to false automatically for the duration of the upgrade to avoid upgrade issues.

Prefixless replication with the IdentityReplicationPolicy

Full support, including replication monitoring, is introduced for the IdentityReplicationPolicy. Unlike the DefaultReplicationPolicy, this policy does not rename remote (replicated) topics on target clusters. That is, the topics that you replicate will have the same name in both source and target clusters. This replication policy is recommended for deployments where SRM is used to aggregate data from multiple streaming pipelines. Alternatively, this replication policy can also be used if the deployment requires MirrorMaker1 (MM1) compatible replication.

Prefixless replication is enabled in Cloudera Manager with the Enable Prefixless Replication property. This property configures SRM to use the IdentityReplicationPolicy and enables internal topic based remote topic discovery, which is required for replication monitoring.

Limitations:

Replication loop detection is not supported. As a result, you must ensure that topics are not replicated in a loop between your source and target clusters.
The /v2/topic-metrics/{target}/{downstreamTopic}/{metric} endpoint of SRM Service v2 API does not work properly with prefixless replication. Use the /v2/topic-metrics/{source}/{target}/{upstreamTopic}/{metric} endpoint instead.
The replication metric graphs shown on the Topic Details page of the SMM UI do not work with prefixless replication.

For more information, see

Internal topic based remote topic discovery

From now on, SRM uses an internal Kafka topic to keep track of remote (replicated) topics. Previously, SRM relied on the naming conventions (prefixes) used by the DefaultReplicationPolicy to discover and track remote topics.

This feature enables SRM to provide better monitoring insights on replications. Additionally, if the feature is enabled, SRM is capable of providing replication monitoring even if a replication policy other than the DefaultReplicationPolicy is in use. Most notably, this enables replication monitoring when SRM is configured for prefixless replication with the IdentityReplicationPolicy.

This feature is enabled in Cloudera Manager by selecting the Remote Topics Discovery With Internal Topic property. The property is selected by default on newly deployed clusters, but must be enabled manually for existing clusters after an upgrade. Cloudera recommends that you enable this feature no matter what replication policy you are using.

For more information, see Streams Replication Manager remote topic discovery .

Configurations to customize replication-records-lag metric calculation

Three new properties are introduced that enable you to control how SRM calculates the replication-records-lag metric. This metric provides information regarding the replication lag based on offsets. The metric is available both on the cluster and the topic level. The following new properties are introduced because the calculation of the metric with default configurations might add latency to replications and impact SRM performance. While these properties are configured in Cloudera Manager, they do not have dedicated configuration entries. Instead, you add them to Streams Replication Manager's Replication Configs to configure them.

Table 1.
Property	Default Value	Description
`replication.records.lag.calc.enabled`	`true`	Controls whether the `replication-records-lag` metric is calculated. This metric provides information regarding the replication lag based on offsets. The metric is available both on the cluster and the topic level. The calculation of this metric might add latency to replications and impact SRM performance. If you are experiencing performance issues, you can try setting this property to false to disable the calculation of `replication-records-lag`. Alternatively, you can try fine-tuning how SRM calculates `replication-records-lag` with the `replication.records.lag.calc.period.ms` and `replication.records.lag.end.offset.timeout.ms` properties.
`replication.records.lag.calc.period.ms`	0	Controls how frequently SRM calculates the `replication-records-lag` metric. The default value of 0 means that the metric is calculated continuously. Cloudera recommends configuring this property to 15000 ms (15 seconds) or higher if you are experiencing issues related to the calculation of `replication-records-lag`. A calculation frequency of 15 seconds or more results in the metric being available for consumption without any significant impact on SRM performance.
`replication.records.lag.end.offset.timeout.ms`	6000	Specifies the Kafka end offset timeout value used for calculating the `replication-records-lag` metric. Setting this property to a lower value than the default 6000 ms (1 minute) might reduce latency in calculating `replication-records-lag`, however, replication-records-lag calculation might fail. A value higher than the default can help avoid metric calculation failures, but might increase replication latency and lower SRM performance.