Cloudera Docs

What's new in Streams Messaging Manager

Learn about the new features for Streams Messaging Manager in Cloudera Runtime 7.1.8.

Improvement in the Connect tab of the SMM UI

You can now deploy Kafka Connect connector configurations containing secret properties which will be stored in an encrypted storage (by default in Kafka). The deployed configuration will only contain references to these secrets. With this comes the need to mark properties as secret on the Streams Messaging Manager user interface so a new connector creation form is introduced, which supports it. You can import configurations and populate the form automatically.

Data Explorer Performance Optimization

The Data Explorer performance has been improved. The SMM UI has the following changes:

  • 'null' values are shown explicitly as null (in italics to signify its not the string literal 'null').
  • Message keys can now be truncated as well and have a 'show more' button.
  • When the 'show more' button is clicked the non-truncated key/value if fetched.
  • Truncated content <1MB are opened in a modal, otherwise a new tab is opened and the content is written to it.
  • Numerical values >9007199254740991 are now shown correctly.

Kafka Connect improvement

  • In NiFi connectors you can now provide file path or URL for the flow.snapshot or alternatively you can browse the file in the SMM UI and then populate the flow.snapshot property with the content of that file.
  • You can now import Connector Configurations as a whole instead of adding individual configurations.
  • Connector configuration validation errors are now correlated with individual config key.
  • Sensitive properties are now hidden from the SMM UI and support is added to set properties as sensitive.
  • To monitor and manage Kafka Connect in SMM, a number of SMM service properties must be configured. From now on, these properties are automatically configured if Kafka Connect Host is left empty (default). This means that the SMM service automatically configures itself to connect to its co-located Kafka Connect instance. You can provide custom values for all properties if you want to override the defaults. The automatically configured properties are as follows:
    • Kafka Connect Host
    • Kafka Connect Port
    • Kafka Connect Protocol

Partition dimension removal in SMM

The partition dimensions of the producer ("/api/v2/admin/metrics/aggregated/producers") and consumer ("/api/v2/admin/metrics/aggregated/groups") metrics are removed from the SMM cache, and are not exposed anymore through the API. This made the SMM memory footprint smaller, relieved some of the load from the metric store, and the network traffic became smaller. With this change, you get a cleaner, and easily readable API, and the UI is snappier, and faster than before.

The version of the /api/v1/admin/metrics/aggregated/* and /api/v1/admin/lineage/* endpoints have been changed to /api/v2/admin/aggregated and /api/v2/admin/lineage. With this change, the response objects are changed as well.

For the /lineage endpoints a common lineage response object is introduced in v2 as opposed to the specific (and different) objects in the experimental v1 endpoint.

For the /aggregated/* endpoints, the partition level metrics (that were in the wrappedPartitionMetrics field) are removed. Partition level metrics have been removed from the /aggregate/producers and /aggregated/producers/{producerClientId} but they are still available in the corresponding /metrics/producers and /metrics/producers/{producerId} endpoints.

Ability to reset consumer group offsets in SMM

Users are now able to reset consumer group offsets so that the consumers will consume messages again. This is especially helpful when a bulk data import operation fails and needs to be retried.

Stateless Sink and Source should populate Key/Value Converters

SMM UI Connector Creation page now contains a default key/value converter to the StatelessNiFiSource or StatelessNiFiSink connectors.

SMM UI shows the replication status tooltip

Streams Messaging Manager now shows tooltip for the replication status.

On the Overview page the lineage information shown is adjusted

On the Overview page, when a Producer or a Consumer is selected, an arrow points to the topic(s) it produced to or consumed from, instead of the partitions.

SMM: HA Support for Oracle Database with RAC

Added HA support for Oracle database with RAC for SMM.

Added API to enrich a connector sample configuration

Streams Messaging Manager API /connector-templates/config/enhance is added, which accepts a sample connector configuration and enhances it with the properties that are probably needed for that connector.

Added emit.consumer.metrics config to SMM CSD, and removed unused SMON host/port configs

Removed "cm.metrics.service.monitor.host" and "cm.metrics.service.monitor.port" configurations from Streams Messaging Manager.

These no longer have to be configured as SMM automatically detects ServiceMonitor's location and emits the ConsumerGroup metrics into it.

Added "emit.consumer.metrics" configuration to Streams Messaging Manager.

In case this flag is disabled, Streams Messaging Manager does not emit historic ConsumerGroup metrics into ServiceMonitor, meaning historic metrics (for group Lag and CommittedOffset) would not be available for Groups in SMM. These metrics are used to populate the charts at the bottom of the ConsumerGroupDetail page, or accessed through the "api/v2/admin/metrics/consumers/group/{groupId}" REST API endpoint.

The SMM API now hides email notifier SMTP passwords in its response

Previously the /notifiers endpoint returned the full configuration of the notifier. In the case of email notifiers, the configuration included the password of the SMTP server. API responses from now on do not include the password. As result of this change, the PASSWORD field of existing email notifiers is left blank when you edit them. If you decide to edit the notifier you must reenter the password.

Added common parameters for Stateless NiFi Connectors in SMM

The following new properties are added to SMM to support Stateless NiFi Connectors. Users can add these to the connector config json manually any time.
Property Description Default
extensions.directory Specifies the directory that stores the extensions that will be downloaded (if any) from the configured Extension Client. /tmp/nifi-stateless-extensions
working.directory Specifies the temporary working directory for expanding NiFi Archives (NARs). /tmp/nifi-stateless-working
krb5.file Specifies the krb5.conf file to use if connecting to Kerberos-enabled services. /etc/krb5.conf
nexus.url Specifies the Base URL of the Nexus instance to source extensions from.
flow.definitions.directory Where the flow definitions are stored.