What's New in Streams Replication Manager
Learn about the new features of Streams Replication Manager in Cloudera Runtime 7.1.8.
New property Metric Reporting Period
A new property, Metric Reporting Period
metrics.period), is introduced for the SRM service. This
property configures the frequency (in seconds) at which metrics should be reported.
New properties related metrics processor batching
- SRM Service Streams Cache Maximum Bytes Buffering
- SRM Service Streams Commit Interval
Internal SRM topics are now automatically added to the replication deny list
SRM now ignores internal SRM topics when replicating data. If required, this behavior can
be disabled by adding
Streams Replication Manager's Replication Configs in Cloudera
Log format changes
The SRM Log Format property was moved to the service level from the SRM Service role level, and now applies to both SRM Service and SRM Driver. Changes made to the SRM Log Formatproperty before upgrading to the new version are lost and need to be applied again after the upgrade. The name of the log files of the SRM Driver and SRM Service will contain the host name. Additionally, the SRM Service now accepts all log configurations specified on the Cloudera Manager UI, and does a size-based rotation of the log files instead of a time based rotation.
SRM Service new intra cluster protocol and internal changelog data format
- Use Legacy Intra Cluster Host Name Format
- Use Legacy Internal Changelog Data
New properties for SRM explicit topic creation
SRM now supports user configured values for partition count and/or replication factor of it's internal topics. Multiple dedicated configurations are introduced:
- Metrics Topics Partition Count (
- SRM Control Topics Replication Factor
- SRM Service Advertisement Topics Replication Factor For Remote Queries
- Streams Applications Internal Topics Replication Factor
Improved error message for the
srm-control tool when
SECURESTOREPASS is not set
The error message that is displayed when the
variable is not set for the
srm-control tool has been improved. Instead of
failing with a
NullPointerException, an error message was added that
notifies users that the
SECURESTOREPASS is not configured.
SRM Service Remote Querying
A new feature, Remote Querying, is introduced for the SRM Service. Remote Querying enables the SRM Service to query other, remote SRM Services and fetch the metrics gathered by the remote SRM Services. This allows users to monitor all replications of a deployment that has multiple instances of SRM with a single SRM Service. For more information, see Remote Querying and Configuring Remote Querying.
SRM Service Multi Target Cluster Support
The SRM Service can now target and monitor multiple Kafka clusters. Previously, the Service could only target a single cluster. Targeting multiple clusters enables you to monitor multiple remote Kafka clusters using a single SRM deployment. Configuration is done in Cloudera Manager by adding multiple target Kafka cluster aliases to the Streams Replication Manager Service Target Cluster property. For more information, see Configuring the service role target cluster.
New property, Cluster Alias, introduced for Kafka Credentials
A new optional property, Cluster Alias, is introduced for Kafka Credentials. This property can be used to specify the alias of the Kafka cluster that the Kafka Credential defines. If you choose to use this property, the Name property of a Kafka Credential does not have to match the alias of the Kafka cluster that the Kafka Credential defines. If an alias is not specified in Cluster Alias, the value specified in the Name property is used as the alias.
Increase the default replication factor of internal topics to 3
The internal topics used by SRM are now created with a replication factor of 3 by default. As a result, SRM is now more resistant to host failures. Additionally, Cruise Control can now automatically heal SRM’s internal topics in the event of a single host failure.
SRM now creates all internal topics with correct configurations at startup
The internal topics used by SRM are now automatically created with correct configurations
at startup. These are the metrics topics, the topics used by the
srm-control tool, and the topics used by the SRM Service for service
discovery. Additionally, SRM also verifies that the topics are created with correct
configurations. If the topics are not configured as expected, SRM fails to start. This
improvement fixes CDPD-31745.
Enhance replication status metrics with task information
The SRM Service now reports the status of the Connectors and Tasks in the replication status with detailed descriptions. The status and detailed descriptions can be viewed on the Replications tab of SMM UI.
Auto topic creation is disabled by default on source clusters
A new Connector property,
introduced. This property controls whether auto topic creation is enabled for SRM’s internal
consumers that fetch data from source clusters. This property is enabled by default meaning
that auto topic creation is disabled by default. This property is not directly available for
configuration in Cloudera Manager, and does not have a dedicated configuration entry on the
UI. Instead, it can be configured through the Streams Replication Manager's
Replication Configs Cloudera Manager property. Configuration is possible on a
global or replication level.
SRM configuration properties can be configured globally for Connect workers and Connect connectors
The SRM Driver now accepts configuration properties prefixed with the
connectors. prefixes. Configuration
properties added to Streams Replication Manager's Replication Configs
that use these prefixes are applied globally to all Connect workers or Connect connectors
that the SRM Driver creates. For more information regarding the prefixes, see Understanding SRM properties, their configuration and
hierarchy. For more information on Connect workers and connectors, see Streams Replication Manager Architecture.
SRM Driver monitoring using Cloudera Manager
Cloudera Manager’s ability to monitor the SRM Driver, its replications, and the overall health of SRM is improved. Most notably, the health status of SRM is based on the health of the network and the availability of replication sources and targets. As a result of this improvement, two new metrics, a new health test, and several new configuration properties are introduced for the SRM Driver in Cloudera Manager.
- New metrics and health test
- The new metrics are as follows:
- SRM Driver Distributed Herder Status
- Aggregated Status Code of SRM Driver Replication Flows
The distributed metric describes the status of individual replications. The aggregate metric provides the aggregate status of all replications.
The new health test is called
DISTRIBUTED_HERDER_STATUS. This health test is based on the aggregate metric and provides information on the overall status of SRM and its replications.
- SRM Driver Distributed Herder Status (
- New properties
- The new monitoring related properties are as follows:
- Path for driver plugins (
- Enable HTTP(S) Metrics Reporter
- SSL Encryption for the Metrics Reporter
- HTTP Metrics Reporter Port
- HTTPS Metrics Reporter Port
- Enable Basic Authentication for Metrics Reporter
- Metrics Reporter User Name
- Metrics Reporter Password
- Path for driver plugins (
For more information, see Cloudera Manager Configuration Properties Reference.
SRM Driver now automatically retries setting up replications for unavailable target Kafka clusters
- Retry Count for SRM Driver
- Retry Delay for SRM Driver
For more information see, Cloudera Manager Configuration Properties Reference or Configuring SRM Driver retry behaviour.
Disabled replications can now be fully deactivated by configuring heartbeat emission
As a result of theKafka rebase in this version of Cloudera Runtime, an improvement is introduced in connection with heartbeat emission. From now on, you can fine tune your deployment and fully deactivate any unnecessary replications that are set up by default by configuring heartbeat emission. This can help with minimizing any performance overhead caused by unnecessary replications.
To support this change, an improvement was made for the SRM service in Cloudera Manager. A
dedicated configuration property, Enable Heartbeats, is introduced.
You can use this property to configure
emit.heartbeats.enabled on a global
level directly in Cloudera Manager. Replication level overrides are still supported. This
can be done by adding
emit.heartbeats.enabled with a valid replication
prefix to Streams Replication Manager's Replication Configs. For more
information on configuring heartbeat emission, see Configuring SRM Driver heartbeat emission.
IdentityReplicationPolicy now available
IdentityReplicationPolicyis available for use with Streams Replication Manager. This replication policy does not rename remote (replicated) topics. Streams Replication Manager can be configured to use this replication policy by adding the following entry to Streams Replication Manager's Replication Configs:
For more information, see KAFKA-9726.
SRM Service Basic Authentication support
The SRM Service can now be secured using Basic Authentication. Once Basic Authentication is set up and enabled, the REST API of the SRM Service becomes secured. Any clients or services that connect to the REST API will be required to present valid credentials for access. Configuration is done in Cloudera Manager using SRM configuration properties and external accounts. For more information, see Configuring Basic Authentication for the SRM Service.
SRM automatically creates a Basic Authentication credential for co-located services
SRM automatically creates a Basic Authentication credential for co-located services (you can change the credentials using SRM Service Co-Located Service Username and SRM Service Co-Located Service User Password). When Basic Authentication is enabled, this user is automatically accepted by the SRM Service. For more information, see Configuring Basic Authentication for the SRM Service.