Cloudera Docs

What's new in Streams Messaging Manager

Learn about the new features of Streams Messaging Manager in Cloudera Runtime 7.2.18.

UI updates

Various improvements and new features are introduced for the SMM UI. The notable changes are as follows:
Cruise Control UI

A new page is added to Streams Messaging Manager to monitor the Kafka cluster state and rebalancing process with Cruise Control. The Cruise Control User Interface (UI) enables you to review and configure the rebalancing of Kafka clusters through dashboards and a rebalancing wizard. The available goals and anomaly detectors are based on the Cloudera Manager configurations of Cruise Control. You can access Cruise Control from SMM using the on the navigation sidebar.

For more information about Cruise Control in SMM, see Monitoring and managing Kafka cluster rebalancing.

Data Explorer
  • When you view Avro data in the Data Explorer, logicalTypes are converted by default. That is, instead of showing the underlying type, (for example, byte) the Data Explorer displays proper deserialized values.
  • Avro messages are now pretty printed when you open them using the Show More option.
Kafka Connect
  • Hovering over the status icons of connector tasks now displays the status text instead of the name of the icon.
  • The Add missing configurations option now populates missing properties with default values.
  • Adding flow.snapshot into a key field of a password type property clears password placeholders.
  • The value field of the flow.snapshot property is now always a text area. Previously, if the property was added manually, the value field was a text field instead of an area.
  • Text found in the (Help) tooltip of connector property values now displays properly. Long strings no longer overflow the tooltip. Additionally, property descriptions are truncated. Clicking or more... displays a pop-up containing the full description.
  • Validating a connector configuration when the Kafka service is stopped returns a Connection refused error instead of validation passing.
  • Deploying a new connector that has the same name as an existing connector no longer updates the existing connector. Instead, the connector deployment fails with Connector [***NAME***] already exists.
  • A loading animation is displayed when loading connector templates.
  • The horizontal divider found in the context menu of properties is no longer displayed if the String , Boolean, Number, and Password options are not available for the property

For more information regarding the various new features and options related to Kafka Connect, see Managing and monitoring Kafka Connect using Streams Messaging Manager.

Other
All charts present on the UI received a visual update. Additionally, more details are presented about the data they display.

Changes in Prometheus setup and configuration

Kafka Connect is now capable of securing its metric reporter with TLS/SSL and Basic Authentication. As a result, the setup steps required to configure Prometheus as the metrics store for SMM are changed. For updated deployment instructions, see Setting up Prometheus for Streams Messaging Manager.

SMM internal Kafka topics are created with a replication factor of 3

From now on the __smm* internal SMM topics are created with a replication factor of 3. This change is only true for newly deployed clusters. The replication factor is not updated during the upgrade. Cloudera recommends that you increase the replication factor of these topics to 3 with kafka-reassign-partitions following an upgrade.

Remove keystore from SMM Schema Registry client configuration if Kerberos is enabled for Schema Registry

SMM uses a Schema Registry client to fetch schemas from Schema Registry. This Schema Registry client has Kerberos authentication properties and keystore properties for mTLS. Typically, the Schema Registry server, by default, does not allow mTLS authentication. But if mTLS is enabled in the Schema Registry server, then mTLS authentication has a higher precedence than Kerberos. Therefore, the mTLS principal (from the keystore) is used for authorization with Ranger rather than the Kerberos principal. This might result in authorization failures if the mTLS principal is not added to Ranger to access the Schema Registry resources.

From now on, the Schema Registry client used by SMM does not have keystore properties for mTLS when Kerberos is enabled. As a result, even if mTLS is enabled for the Schema Registry server, the Kerberos principal is used for authentication and authorization with Ranger.

Dedicated endpoint for connector creation

A dedicated endpoint for connector creation is introduced. The endpoint is POST /api/v1/admin/kafka-connect/connectors. Requests made to this endpoint fail with the following error message if the connector name specified in the request exists.
{
  "error_code": 409,
  "message": "Connector [***NAME***] already exists"
}

In previous versions, PUT api/v1/admin/kafka-connect/connectors/{connector} was the only endpoint you could use to create connectors. However, this endpoint is also used to update connectors. If you specify the name of an existing connector in the request, the endpoint updates existing connector. As a result, Cloudera recommends that you use the new endpoint for connector creation going forward. The SMM UI is also updated and uses the new endpoint for connector creation.

Jersey client timeout now configurable

SMM uses internal Jersey clients to make requests to Kafka Connect and Cruise Control. The connection and read timeouts for these clients was previously hard-coded to 30 seconds. Configuring them was not possible. This release introduces new properties, which enable you to configure the connection and read timeouts of these clients. The default timeout remains 30 seconds. The properties introduced are as follows.

  • Kafka Connect Client Connect timeout
  • Kafka Connect Client Read timeout
  • Cruise Control Client Connect timeout
  • Cruise Control Client Read timeout