What's new in Streams Messaging Manager

Learn about the new features of Streams Messaging Manager in Cloudera Runtime 7.2.14.

Reactive Lineage fetching from Kafka producer cache

You can now visualize the lineage between producers and consumers in SMM. Lineage information helps you to understand how the message is moving from a producer to a consumer group and which topics or partitions are part of that flow. Lineage between clients and topics or partitions are now shown using the new lineage endpoints. For more information, see Monitoring lineage information.

New endpoint added to fetch lineage information for a topic

The /api/v1/admin/lineage/partitions/{topic} endpoint used to fetch which producers have produced into the queried topic, and which consumerGroup's members have consumed from it. Now when you click on a topic to fetch the lineage on the UI, this endpoint is used.

New endpoint added to fetch lineage information for a consumerGroup

The /api/v1/admin/lineage/consumerGroups/{consumerGroupId} endpoint used to fetch which topics the members of that consumerGroup have consumed from, and also what producers have produced into those topics. Now when you click on a group on the UI to fetch the lineage, this endpoint is used.

New endpoint added to fetch lineage information for a topicPartition

The /api/v1/admin/lineage/partitions/{topic}/{partition} endpoint is used to fetch which producers have produced into that queried topicPartition and which consumerGroup members have consumed from that topicPartition. Now when you click on a topicPartition to fetch the lineage on the UI, this endpoint is used.

New endpoint added to fetch lineage information for a producer

The /api/v1/admin/lineage/lineage/producers/{producerId} endpoint is used to fetch which topics the queried producer has produced into, and which consumerGroups members have consumed from those topics. Now when you click on a producer to get the lineage on the UI, this endpoint is used.

On selecting the partition on Overview page, the new lineage endpoint should be called

Lineage between clients and topics or partitions are now shown using the new lineage endpoints. Remember that when checking the lineage (connected clients) for a TopicPartition, only the recently connected clients will be shown.

Support added for multiple replication targets in SMM

SMM now supports SRM replication flows targeting remote clusters making use of the new v2 SRM APIs.

Remote Replication flows available under the “/api/v2/admin/replication-stats” APIs. The UI is now configured to make use of these new APIs.

Introduced multi-target replication monitoring support in alerts

• SMM now adopted the new V2 SRM Service endpoints upon which alerting is based on.

• In SMM when configuring alerts for replications in the UI now source and target clusters can be defined, as opposed to the previous configuring panel, where only the source cluster could be defined (since the target cluster was fixed to be the colocated Kafka cluster).

• Old alerts will still function, however editing them can only be done using the new format, where source and target clusters have to be defined.

• IMPORTANT: For alerts involving remote SRM cluster queries set the execution interval to at the very least a minute (preferably more).

SMM authenticates to SRM Service

SMM now automatically configures Basic Authentication when connecting to SRM and the service dependency based auto-configuration is in use.

For manual SRM connectivity configurations, Basic Auth configurations were added (Streams Replication Manager Basic Authentication, Streams Replication Manager Basic Authentication Username, Streams Replication Manager Basic Authentication Password).

SMM Cache-Control is part of default SMM REST Server API's responses' headers

The new SMM configuration named cache.control.http.response.header.value allows to configure the Cache-Control header's value for certain endpoints. Configure it in the following key-value like fashion:

• The key is the path prefix to the endpoints where the Cache-Control header should be added.
• The value is the value of Cache-Control header.

In order to turn off functionalities provided by the Cache-Control header just delete the entries, or set the value to no-store.

Added helper tooltips to SMM UI

SMM now provides more informative tooltips (hover over the table headers and labels) for most of its elements in the web UI.

Removed Consumer Rate graphs

The lag rate graph is removed from the UI.

The lag rate values are removed from the /api/v1/admin/metrics/aggregated/groups/{groupName} and /api/v1/admin/metrics/aggregated/groups endpoints.

SMM is automatically integrated with co-located Kafka Connect

To monitor and manage Kafka Connect in SMM, a number of SMM service properties must be configured. These are the following:

• Kafka Connect Host
• Kafka Connect Port
• Kafka Connect Protocol

From now on, these properties are automatically configured if Kafka Connect Host is left empty (default). This means that the SMM service automatically configures itself to connect to its co-located Kafka Connect instance. You can provide custom values for all properties if you want to override the defaults.