Cloudera Docs

Behavioral Changes in Streams Messaging

Review the list of Streams Messaging behavioral changes in Cloudera DataFlow for Data Hub 7.3.2.

Cloudera DataFlow for Data Hub 7.3.2 introduces Streams Messaging functional adjustments and behavioral updates, and includes all service packs and cumulative hotfixes from Cloudera Runtime 7.3.1.100 through 7.3.1.706. For a comprehensive record of all Streams Messaging functional adjustments in Cloudera Runtime 7.3.1.x, see Behavioral Changes.

Behavioral Changes in Kafka

Functional adjustments and behavioral updates for Kafka are introduced in Cloudera DataFlow for Data Hub 7.3.2, its service packs, and cumulative hotfixes.

7.3.2

Component-level custom Java home configuration removed
Previous behavior:

You could configure a component-specific Java home for Kafka Connect.

New behavior:

The component-level custom Java home configuration options are removed. Kafka Connect now uses the host-level java_home configuration. If you previously set a component-specific Java home for this service, verify the host-level java_home setting after upgrading.

High Watermark no longer advances when ISR is below MinISR
Previous behavior:

The High Watermark (HWM) advanced regardless of whether the in-sync replica (ISR) count was below min.insync.replicas. When producers used acks=1 or acks=0, messages were written to the leader and became consumable once the HWM advanced, even if the ISR had dropped below the minimum threshold. The min.insync.replicas setting only affected acks=all produce requests, blocking writes when ISR was insufficient, but did not prevent HWM advancement or consumer reads for acks=0/acks=1 messages.

New behavior:

The HWM no longer advances when the ISR count falls below min.insync.replicas. As a result, consumers are blocked from reading new messages in this condition, even if producers with acks=1 or acks=0 are still writing to the leader. This ensures data is only consumable when it meets the cluster's minimum durability requirements. If you use min.insync.replicas=2 or higher, you may see reduced consumer throughput when the ISR count drops below the configured minimum.

For more information, see KIP-966: Eligible Leader Replicas and KAFKA-15583.

Kafka protocol version is set automatically during upgrades
Previous behavior:

The inter.broker.protocol.version property for ZooKeeper-based clusters and the metadata.version property for KRaft-based clusters were not set automatically before an upgrade. Manually configuring these properties to the current protocol and metadata version was required before an upgrade.

New behavior:

During a cluster upgrade, Cloudera Manager now automatically sets the inter.broker.protocol.version property for ZooKeeper-based clusters and the metadata.version property for KRaft-based clusters. Manual configuration is no longer required.

Behavioral Changes in Schema Registry

Functional adjustments and behavioral updates for Schema Registry are introduced in Cloudera DataFlow for Data Hub 7.3.2, its service packs, and cumulative hotfixes.

7.3.2

Component-level custom Java home configuration removed
Previous behavior:

You could configure a component-specific Java home for Schema Registry.

New behavior:

The component-level custom Java home configuration options are removed. Schema Registry now uses the host-level java_home configuration. If you previously set a component-specific Java home for this service, verify the host-level java_home setting after upgrading.

Schema Registry now defaults to IPv4-only communication

The default value of the schema.registry.additional.java.options configuration parameter was updated to set the IP protocol to IPv4.

If you changed the default value of this parameter before upgrading, the new default value is not applied on upgrade. You can apply it manually after the upgrade.

Behavioral Changes in Streams Messaging Manager

Functional adjustments and behavioral updates for Streams Messaging Manager are introduced in Cloudera DataFlow for Data Hub 7.3.2, its service packs, and cumulative hotfixes.

7.3.2

Component-level custom Java home configuration removed
Previous behavior:

You could configure a component-specific Java home for Streams Messaging Manager.

New behavior:

The component-level custom Java home configuration options are removed. Streams Messaging Manager now uses the host-level java_home configuration. If you previously set a component-specific Java home for this service, verify the host-level java_home setting after upgrading.

Default JMX settings changed to restrict connections to localhost
Previous behavior:

The default value of the SMM_JMX_OPTS Cloudera Manager configuration option was -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false, which allowed unrestricted, unauthenticated JMX connections.

New behavior:

The default value of SMM_JMX_OPTS has been changed to restrict JMX connections to localhost only and enable SSL. If you previously customized SMM_JMX_OPTS, your custom value is preserved on upgrade. To revert to open JMX, update SMM_JMX_OPTS in Cloudera Manager.

Streams Messaging Manager now defaults to IPv4-only communication

A new argument was added to SMM_JVM_PERF_OPTS that sets the IP protocol to IPv4 by default.

If you changed the default value of this parameter before upgrading, the new default value is not applied on upgrade. You can apply it manually after the upgrade.

Streams Messaging Manager UI Migration to Java
The Streams Messaging Manager UI service is migrated from a NodeJS runtime to a Java-based server. This change addresses security vulnerabilities associated with NodeJS dependencies and aligns Streams Messaging Manager with the centralized dependency management of the platform.
As a result of this migration, the following changes apply:
  • Runtime environment

    The Streams Messaging Manager UI service now runs on the JVM. Configuration for the runtime environment is now managed via the SMM_JAVA_OPTS environment variable.

  • TLS configuration
    TLS configuration moved from OpenSSL-style parameters to standard Java JSSE configuration. New Cloudera Manager parameters manage TLS protocols and cipher suites:
    • streams.messaging.manager.ui.ssl.supportedCipherSuites
    • streams.messaging.manager.ui.ssl.excludedCipherSuites
    • streams.messaging.manager.ui.ssl.supportedProtocols
    • streams.messaging.manager.ui.ssl.excludedProtocols

  • Configuration migration

    During upgrade, Cloudera Manager attempts to automatically migrate existing TLS settings (including those found in the NODE_OPTIONS environment variable within safety valves) to the new Java-based configuration parameters. However, manual verification is strongly recommended.

  • Safety valves

    Any properties previously set in the Streams Messaging Manager UI Server Environment Advanced configuration Snippet (Safety Valve) using NODE_OPTIONS that are not related to TLS must be manually translated to their Java equivalents (if applicable) and set using SMM_JAVA_OPTS.

Behavioral Changes in Streams Replication Manager

Functional adjustments and behavioral updates for Streams Replication Manager are introduced in Cloudera DataFlow for Data Hub 7.3.2, its service packs, and cumulative hotfixes.

7.3.2

Component-level custom Java home configuration removed
Previous behavior:

You could configure a component-specific Java home for Streams Replication Manager.

New behavior:

The component-level custom Java home configuration options are removed. Streams Replication Manager now uses the host-level java_home configuration. If you previously set a component-specific Java home for this service, verify the host-level java_home setting after upgrading.

Default JMX settings changed to restrict connections to localhost
Previous behavior:

The default value of the SRM_JMX_OPTS Cloudera Manager configuration option was -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false, which allowed unrestricted, unauthenticated JMX connections.

New behavior:

The default value of SRM_JMX_OPTS has been changed to restrict JMX connections to localhost only and enable SSL. If you previously customized SRM_JMX_OPTS, your custom value is preserved on upgrade. To revert to open JMX, update SRM_JMX_OPTS in Cloudera Manager.

Public Kafka Connect endpoints removed from SRM REST server

The public Kafka Connect endpoints are removed from the SRM REST server. Previously, these endpoints allowed users to interact with the internal Kafka Connect cluster inside the SRM instance, including starting arbitrary connectors and modifying internal connectors. These endpoints were undocumented and not part of the official SRM API. They are no longer available.

Streams Replication Manager now defaults to IPv4-only communication

A new argument was added to SRM_JVM_PERF_OPTS that sets the IP protocol to IPv4 by default.

If you changed the default value of this parameter before upgrading, the new default value is not applied on upgrade. You can apply it manually after the upgrade.

Change in internal topic filtering logic
Summary:

The logic that identifies and filters internal topics in Streams Replication Manager has changed. This enables the replication of topics that appear to be internal but are not truly internal to Kafka and Streams Replication Manager, reducing the risk of unintentionally excluding user topics from replication.

Previous behavior:
Topics were filtered from replication by the DefaultReplicationPolicy and IdentityReplicationPolicy policies if the topic name ended with [***SEPARATOR***]internal. For example, .internal. In addition, the default deny list regex pattern was the following:
.*[\\-\\.]internal, .*\\.replica, __.*
New behavior:
Topics are now filtered from replication by the DefaultReplicationPolicy and IdentityReplicationPolicy policies if:
  • Name starts with mm2 and ends with [***SEPARATOR***]internal
  • Name ends with [***SEPARATOR***]checkpoints[***SEPARATOR***]internal
In addition, the default deny list regex pattern is now the following:
mm2.*\\.internal, .*\\.replica, __.*
Internal Kafka topics that start with a dot (.) or two underscores (__) are continued to be filtered from replication by default.

Behavioral Changes in Cruise Control

Functional adjustments and behavioral updates for Cruise Control are introduced in Cloudera DataFlow for Data Hub 7.3.2, its service packs, and cumulative hotfixes.

7.3.2

Summary: ZooKeeper service dependency removed from Cruise Control
Previous behavior:

Cruise Control had a ZooKeeper service dependency in Cloudera Manager.

New behavior:

The ZooKeeper service dependency is removed.

Summary: Default JMX settings changed to restrict connections to localhost
Previous behavior:

No dedicated Cloudera Manager configuration option existed to control JMX JVM flags for Cruise Control. JMX connections were unrestricted by default.

New behavior:

A new CC_JMX_OPTS Cloudera Manager configuration option is available. Its default value restricts JMX connections to localhost only and enables SSL. If you previously set CC_JMX_OPTS, your custom value is preserved on upgrade. To revert to open JMX, update CC_JMX_OPTS in Cloudera Manager.

Summary: Default Supported Goals and Anomaly Detection Goals updated in Cloudera Manager
Previous behavior:

The Cloudera Manager defaults for Supported Goals did not include BrokerSetAwareGoal. The defaults for Anomaly Detection Goals covered rack, replica, and disk capacity only, not network inbound, network outbound, or CPU capacity goals.

New behavior:

The default Supported Goals list includes BrokerSetAwareGoal. The default Anomaly Detection Goals list also includes NetworkInboundCapacityGoal, NetworkOutboundCapacityGoal, and CpuCapacityGoal. Cloudera Manager defaults for these goal lists now match the Cruise Control defaults. No action is required after upgrade.