Cloudera Docs

What's new in Streams Messaging Manager

Learn about the new features for Streams Messaging Manager in Cloudera Runtime 7.1.9.

UI updates

Various improvements are introduced for the SMM UI. The notable changes are as follows:
Data Explorer
  • When you view Avro data in the Data Explorer, logicalTypes are converted by default. That is, instead of showing the underlying type, (for example, byte) the Data Explorer displays proper deserialized values.
  • Avro messages are now pretty printed when you open them using the Show More option.
  • The modal window that you use to view messages now includes a copy to clipboard button if the message you are viewing is long.
  • A (Refresh) option is added next to the FROM OFFSET field. This option refreshes the partition offset range and fetches the latest messages.
  • You can now view JSON output in pretty printed format by selecting the JSON Pretty Print deserializer.
  • A new drop-down, ISOLATION LEVEL:, is added to the Data Explorer.

    The drop-down configures the isolation.level property of the Kafka consumer that the Data Explorer uses. The isolation level is also configurable when using the REST API with the consumerIsolationLevel parameter of the /api/v1/admin/topics/{topicName}/partition/{partitionId}/payloads endpoint. The accepted values are read_committed and read_uncommitted. The default value is read_uncommitted.

Kafka Connect
  • Hovering over the status icons of connector tasks now displays the status text instead of the name of the icon.
  • The Add missing configurations option now populates missing properties with default values.
  • Adding flow.snapshot into a key field of a password type property clears password placeholders.
  • An error page is displayed if you navigate to a connector that does not exist.
  • A new option, Add, is added to the Import a Connector config… modal. This option enables you to import connector configuration properties without overriding existing properties.
  • Search and autocomplete are now available for connector property keys. In addition, you can now filter property keys based on their group and importance.
  • An error message is added that notifies you if validation errors are found for properties that are currently filtered.
  • A Reset Filters option is added. This option resets all search filters.
  • Three new actions are added that modify the configuration as a whole. The options are Remove all, Reset, and Export. These actions are available in a new Actions drop-down.
  • The Import Connector Configuration… option is moved to the Actions drop-down and is renamed to Import.
  • The Deployment Status modal now correctly displays the status of the deployment process.
  • If available, the display names of configuration property keys are displayed above the property key.
  • The Connector Setup wizard is updated. Connector (template) selection and connector configuration now happen on separate pages of the wizard.
  • A (Help) option is added that provides detailed information about each property key. The icon is only available for properties that have their metadata (description, type, group, and so on) defined.
  • Types can be specified for properties.

For more information regarding the various new features and options related to Kafka Connect, see Managing and monitoring Kafka Connect using Streams Messaging Manager.

Other
  • The style of SMM UI is updated. This update includes various changes to the colors, fonts, and overall style of the UI.
  • You can now increase the number of partitions of a topic. The option is available on the CONFIGS tab on the Topic Details page. Decreasing the partitions of a topic is not possible.
  • The SMM UI contains Brokers and Topics pages where records contain broker or topic specific partition lists and their profile pages as well. All partition list columns are now sortable.
  • Log-size related information is now displayed about brokers, topics, and partitions. Furthermore, warning messages appear when log directory related errors are reported by Kafka.
  • The Cluster Replications tab now also shows the replication-records-lag metric.

Changes in Prometheus setup and configuration

In this version of Cloudera Runtime, a number of improvements are introduced related to how Kafka Connect exposes its metrics. Most notably:

  • Kafka Connect is now capable of securing its metric reporter with TLS/SSL and Basic Authentication
  • The default port of the Kafka Connect metric reporter is changed from 28084 to 28806 (unsecure) or 28807 (secure).
  • The Kafka Connect Prometheus Metrics Port property is removed and is replaced by Secure Jetty Metrics Port and Jetty Metrics Port.

As a result, the setup steps required to configure Prometheus as the metrics store for SMM are changed. For updated deployment instructions, see Setting up Prometheus for Streams Messaging Manager.

If you already use Prometheus with SMM, you must make changes to your Prometheus configuration following a cluster upgrade and update the Kafka Connect port that Prometheus connects to. If configuration is not done, Kafka Connect metrics will no longer be available in SMM. For exact steps, see Step 9: Complete Post-Upgrade steps for upgrades to CDP Private Cloud Base.

SMM supports connections to databases secured using TLS 1.2 and TCPS

SMM can connect to TLS-enabled MySQL, MariaDB, or PostgreSQL databases and TCPS-enabled Oracle database. To connect to a TLS/TCPS-enabled database while adding the SMM service to a cluster, see Configure TLS 1.2 for Streams Messaging Manager. You can also enable TLS/TCPS on an existing database and then configure SMM to connect to it. See Set up and configure TLS 1.2 for Streams Messaging Manager. For more information about Oracle TCPS, see How to connect CDP components to a TCPS-enabled Oracle database.

SMM internal Kafka topics are created with a replication factor of 3

From now on the __smm* internal SMM topics are created with a replication factor of 3. This change is only true for newly deployed clusters. The replication factor is not updated during the upgrade. Cloudera recommends that you increase the replication factor of these topics to 3 with kafka-reassign-partitions following an upgrade.

Remove keystore from SMM Schema Registry client configuration if Kerberos is enabled for Schema Registry

SMM uses a Schema Registry client to fetch schemas from Schema Registry. This Schema Registry client has Kerberos authentication properties and keystore properties for mTLS. Typically, the Schema Registry server, by default, does not allow mTLS authentication. But if mTLS is enabled in the Schema Registry server, then mTLS authentication has a higher precedence than Kerberos. Therefore, the mTLS principal (from the keystore) is used for authorization with Ranger rather than the Kerberos principal. This might result in authorization failures if the mTLS principal is not added to Ranger to access the Schema Registry resources.

From now on, the Schema Registry client used by SMM does not have keystore properties for mTLS when Kerberos is enabled. As a result, even if mTLS is enabled for the Schema Registry server, the Kerberos principal is used for authentication and authorization with Ranger.

Highly available Kafka Connect integration

SMM uses the Kafka Connect service role’s REST URL to establish a connection with Connect and to serve Connect metrics. Previously, even if your Connect deployment was highly available and had multiple service roles deployed, SMM could only be configured with a single connection URL. From now on, multiple URLs can be configured. If the Connect service role that SMM is connected to fails, SMM automatically connects to a different instance that is available.

As a result of this change, the Kafka Connect Host and Kafka Connect Port properties are replaced by the Kafka Connect Rest HostPorts property. If Kafka Connect Rest HostPorts is left empty (default), SMM is automatically configured with the host, port, and protocol of the Connect service role instances belonging to the Kafka service selected with the Kafka Service SMM property.

If you previously configured Kafka Connect Host and Kafka Connect Port, the values set in the properties are automatically migrated to Kafka Connect Rest HostPorts when you upgrade.

Partition Assignment tab [Technical Preview]

A new tab,Assignment, is introduced on the Topic Details page. This tab gives you a visual overview of the current state of the partitions and replicas of the topic. Information presented on this page includes various topic-level statistics and the replica assignment of all partitions. If rack awareness is enabled for Kafka, the replica assignment is displayed in a rack-based view. If the rack IDs follow the format of multi-level rack IDs, the rack IDs are rendered as a hierarchy.

In addition to the new tab, the Brokers and Broker Details pages now both show the rack ID of the brokers. Furthermore, a new endpoint, /api/v1/admin/topics/{topicName}/description, is introduced. The endpoint returns information regarding the partitions of a topic.

For more information regarding this tab as well as Kafka rack awareness, see the following resource:

The producer of SMM Kafka interceptors can now be configured

Clients that use either of the SMM monitoring interceptors (MonitoringConsumerInterceptor or MonitoringProducerInterceptor) use a background producer to push client metrics into Kafka every 30 seconds. This background producer can from now on be configured by passing producer configurations to the clients that use the interceptor. Properties are passed to the producer with the smm.monitoring.interceptor.producer.* prefix.

The prefix is trimmed and the remaining part of the configuration is passed to the background producer. For example, if you want to configure the batch.size property for the background producer, you must set the following property: 
smm.monitoring.interceptor.producer.batch.size
If you do not configure the client.id property of the producer, the producer uses smm-monitoring-interceptor as its ID instead of using an empty ID.