What's New in Streams Messaging
Learn about the new Streams Messaging features in Cloudera DataFlow for Data Hub 7.2.6.
- Remove usage of non-FIPS compatible algorithms in Kafka
- Murmur3 hashing is introduced for the log cleaner when it builds offset maps. In addition,
to the introduction of the new hashing algorithm, the default algorithm used is also changed
to Murmur3. The previous default was MD5. If required, MD5 can still be used by adding
cloudera.log.cleaner.hashing.algorithm=MD5to the Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka.properties property in Cloudera Manager.
There are no new features for Schema Registry in this release.
Streams Messaging Manager
- CSD support to configure caching in SMM authorizer
- SMM request processing is sped up by introducing an authorization cache. The default TTL of the cache is 30 seconds and it is configurable in Cloudera Manager. Setting the TTL to 0 disables the cache entirely.
- The affected versions are Cloudera Manager 7.2.4 and higher and CDH 7.2.1 and higher, Cloudera Manager 7.3.0 and higher and CDH 7.1.6 and higher.
- SMM automatically configures SRM in Cloudera Manager
- SMM automatically configures the SRM connection based on a service dependency. Manual configuration options are removed. This feature affects Cloudera Manager versions 7.2.3 and higher and CDH versions 7.2.3 and higher, Cloudera Manager versions 7.3.0 and higher and CDH versions 7.1.6 and higher.
Streams Replication Manager
- SRM is available in Data Hub and CDP Public Cloud
- SRM can now be provisioned in CDP Public Cloud with Data Hub. The default Streams Messaging cluster definitions are updated to include SRM. For more information, see Streams Messaging cluster layout and Creating your first Streams Messaging cluster.
- SRM high availability mode
- You can now deploy SRM in high availability mode. For more information, see Enable high availability for Streams Replication Manager.
- Replication-specific Kafka Connect REST servers
- SRM can now run multiple drivers in the same cluster (high availability). To make this possible, the SRM driver role now deploys a Kafka Connect REST server for each replication that you set up and configure. These REST servers ensure communication between the different instances of the driver role and make replication with multiple drivers in a single cluster possible.
- If required, you can configure these REST servers in Cloudera Manager with the Streams Replication Manager's Replication Configs property and two specific prefixes. For more information, see Configuring replication specific Kafka Connect REST servers .
- The grace and retention period as well as the collection frequency of SRM Service role metrics are configurable
- Configuration properties related to the SRM Service role’s metric processing are
added. These properties give users the ability to configure the grace and retention
periods as well as the collection frequency of SRM Service role metrics. The grace and
retention periods can be configured directly with the following Cloudera Manager properties:
SRM Service Metrics Grace Period (
SRM Service Metrics Retention Period (
metrics.periodproperty can only be configured on a replication level. For example: