What's New in Streams Replication Manager

SRM Driver monitoring using Cloudera Manager🔗

Cloudera Manager’s ability to monitor the SRM Driver, its replications, and the overall health of SRM is improved. Most notably, the health status of SRM is based on the health of the network and the availability of replication sources and targets. As a result of this improvement, two new metrics, a new health test, and several new configuration properties are introduced for the SRM Driver in Cloudera Manager.

New metrics and health test

The new metrics are as follows:

• SRM Driver Distributed Herder Status (streams_replication_manager_distributed_herder_status)
• Aggregated Status Code of SRM Driver Replication Flows (streams_replication_manager_aggregated_herder_status)

The distributed metric describes the status of individual replications. The aggregate metric provides the aggregate status of all replications.

The new health test is called DISTRIBUTED_HERDER_STATUS. This health test is based on the aggregate metric and provides information on the overall status of SRM and its replications.

New properties

The new monitoring related properties are as follows:

• Path for driver plugins (plugin.path)
• Enable HTTP(S) Metrics Reporter (mm.metrics.servlet.enable)
• SSL Encryption for the Metrics Reporter (metrics.jetty.server.ssl.enabled)
• HTTP Metrics Reporter Port (metrics.jetty.server.port)
• HTTPS Metrics Reporter Port (metrics.jetty.server.secureport)
• Enable Basic Authentication for Metrics Reporter (metrics.jetty.server.authentication.enabled)
• Metrics Reporter User Name (metrics.jetty.server.auth.username)
• Metrics Reporter Password (metrics.jetty.server.auth.password)

For more information, see Cloudera Manager Configuration Properties Reference.

SRM Driver now automatically retries setting up replications for unavailable target Kafka clusters🔗

For more information see, Cloudera Manager Configuration Properties Reference or Configuring SRM Driver retry behaviour.

Disabled replications can now be fully deactivated by configuring heartbeat emission🔗

As a result of the rebase to Kafka 2.8 (KAFKA-10710), an improvement is introduced in connection with heartbeat emission. From now on, you can fine tune your deployment and fully deactivate any unnecessary replications that are set up by default by configuring heartbeat emission. This can help with minimizing any performance overhead caused by unnecessary replications.

To support this change, an improvement was made for the SRM service in Cloudera Manager. A dedicated configuration property, Enable Heartbeats, is introduced. You can use this property to configure emit.heartbeats.enabled on a global level directly in Cloudera Manager. Replication level overrides are still supported. This can be done by adding emit.heartbeats.enabled with a valid replication prefix to Streams Replication Manager's Replication Configs. For more information on configuring heartbeat emission, see Configuring SRM Driver heartbeat emission.

IdentityReplicationPolicy now available🔗

replication.policy.class=org.apache.kafka.connect.mirror.IdentityReplicationPolicy

SRM configuration properties can be configured globally for Connect workers and Connect connectors🔗

The SRM Driver now accepts configuration properties prefixed with the workers. and connectors. prefixes. Configuration properties added to Streams Replication Manager's Replication Configs that use these prefixes are applied globally to all Connect workers or Connect connectors that the SRM Driver creates. For more information regarding the prefixes, see Understanding SRM properties, their configuration and hierarchy. For more information on Connect workers and connectors, see Streams Replication Manager Architecture.

SRM Service Basic Authentication support🔗

The SRM Service can now be secured using Basic Authentication. Once Basic Authentication is set up and enabled, the REST API of the SRM Service becomes secured. Any clients or services that connect to the REST API will be required to present valid credentials for access. Configuration is done in Cloudera Manager using SRM configuration properties and external accounts.

SRM automatically creates a Basic Authentication credential for co-located services🔗

SRM automatically creates a Basic Authentication credential for co-located services (users can change the credentials using SRM Service Co-Located Service Username and SRM Service Co-Located Service User Password). When Basic Authentication is enabled, this user is automatically accepted by the SRM Service. For more information, see Configuring Basic Authentication for the SRM Service.

SRM Service Remote Querying no longer in technical preview🔗

SRM Service Remote Querying was introduced in a previous release of Cloudera Runtime as a technical preview feature. Starting with this release, Remote Querying is ready for use in production environments. This is the result of Basic Authentication being introduced for the SRM Service and SMM supporting multi-target alerting.

For more information on Remote Querying, see Remote Querying and Configuring Remote Querying. For more information on how to set up Basic Authentication for Remote Querying, see Configuring Basic Authentication for Remote Querying.

The SRM Driver can now write the origin offset into the record header🔗

SRM now supports a diagnostic feature in which the source offset of the replicated records are written into the headers. The feature can be turned on by setting copy.source.offset.in.header.enabled to true. When enabled, the source offset is written into a header named mm2-source-offset in binary format. The schema of the header payload is available in the connect:mirror-client package, the class name is org.apache.kafka.connect.mirror.SourceOffsets. This feature is only recommended for diagnostic purposes, as the header change increases the size of the replica topic.