What's New in Streams Messaging

Learn about the new Streams Messaging features in Cloudera DataFlow for Data Hub 7.2.9.

Kafka

Kafka metrics endpoints can be secured

TLS/SSL encryption as well as Basic Authentication can now be enabled for the HTTP endpoints of Kafka metrics.

TLS/SSL encryption can be enabled with the SSL Encryption For HTTP Metric Reporter property. However, this property does not take effect on its own, TLS/SSL must be enabled for the Kafka broker for this property to work.

Basic Authentication can be enabled with the Enable HTTP Metric Report Basic Authentication property. In addition to enabling Basic Authentication, you also need to set a username and password used for authentication. These can be configured with the HTTP Metric Report User and HTTP Metric Report Password properties respectively.

For more information regarding these properties, see Kafka Properties in Cloudera Runtime.

Kafka metrics endpoints can be secured

TLS/SSL encryption as well as Basic Authentication can now be enabled for the HTTP endpoints of Kafka metrics.

TLS/SSL encryption can be enabled with the SSL Encryption For HTTP Metric Reporter property. However, this property does not take effect on its own, TLS/SSL must be enabled for the Kafka broker for this property to work.

Basic Authentication can be enabled with the Enable HTTP Metric Report Basic Authentication property. In addition to enabling Basic Authentication, you also need to set a username and password used for authentication. These can be configured with the HTTP Metric Report User and HTTP Metric Report Password properties respectively.

For more information regarding these properties, seeKafka Properties in Cloudera Runtime.

Schema Registry

Import tool for Confluent Schema Registry
You can now import schemas from Confluent Schema Registry's Kafka topic dump. The schemas need to be uploaded as a file to Schema Registry via REST.

The Ranger model was modified and a new permission for "export-import" was added. When authorization is enabled, users need to have the Create permission under "export-import" in order to import the schemas.

For more information, see Importing Confluent Schema Registry schemas into Cloudera Schema Registry.

Streams Messaging Manager

Added timestamp to SMM UI logs

SMM UI logs have timestamps.

Kafka supports producer metric whitelisting

The feature allows producerMetrics to be whitelisted based on a whitelist property that should contain a regex with the allowed client.ids.

In order to enable the feature, set the following properties within Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka.properties/kafka.properties_role_safety_valve:
  • producer.metrics.whitelist.enabled=true
  • producer.metrics.whitelist=regex containing the whitelist for the desired producer's client.ids

This regex uses the simple java java.util.regex.Pattern class to compile the regex, and uses the match() method on the client.id to determine whether it fits the regex or not.

This feature deletes all the producer timeSeries entities from Service Monitor (CM metricsStore) that do not match the regex.

Also this feature filters the metrics within Kafka itself, so producerMetrics that do not match the regex, would not be reported by the /api/metrics endpoint in the Kafka brokers. Both CM and Kafka use the same Java class and method to filter the metrics.

Streams Replication Manager

There are no new features for Streams Replication Manager in this release.