Behavioral Changes in Kafka

Cloudera Runtime 7.3.2.10000 SP1

Kafka Connect Secrets Storage key wrap algorithm is configurable

Previous behavior:

The algorithm that Kafka Connect Secrets Storage uses to wrap the encryption key was hardcoded to AESWrap. You could not change this value in Cloudera Manager.

New behavior:

You can configure the wrap algorithm using the Kafka Connect Secrets Storage Secret Key Wrap Algorithm (kafka.connect.secret.key.wrap.algorithm) property in Cloudera Manager. The default remains AESWrap. On FIPS-enabled clusters where the Bouncy Castle FIPS jars are configured, Cloudera recommends setting this property to AESKWP. Other FIPS-approved algorithms supported by the configured security provider may also be used; AESKWP is the algorithm validated by Cloudera. Changing the wrap algorithm does not re-encrypt secret values stored in the secrets topic. If you change the value from the default and need to run the connect-secret-storage-migration tool, pass kafka.connect.secret.key.wrap.algorithm in the source topic configuration file, similar to kafka.connect.secret.global.password and kafka.connect.secret.pbe.salt.

HTTP metrics SSL enabled by default when Encrypt All Ports is on

Previous behavior:

The default value of SSL Encryption For HTTP Metric Reporter (kafka.http.metrics.ssl.enabled) was false on Kafka Broker and KRaft Controller roles.

New behavior:

When you enable the ENCRYPT_ALL_PORTS feature flag, the effective default becomes true. If you previously set a custom value, your setting is preserved. For information about the feature flag, see Transport Layer Security Support and Custom Protocol Configuration.

Kafka protocol and metadata version is cleared automatically after upgrade

Previous behavior:

After you upgraded a Cloudera on cloud Cloudera Streams Messaging cluster to Cloudera Runtime 7.3.2, you had to manually clear the Kafka Inter-Broker Protocol Version property on ZooKeeper-based clusters or the Kafka Metadata Version property on KRaft-based clusters in Cloudera Manager.

New behavior:

When you upgrade a Cloudera on cloud Cloudera Streams Messaging cluster to Cloudera Runtime 7.3.2 for the first time, Cloudera Manager automatically clears the pinned protocol or metadata version as part of the upgrade.

Service pack upgrades from 7.3.2 to 7.3.2.10000 SP1 do not run that automatic clearing handler. If your cluster was already on 7.3.2 before SP1 and the Kafka Inter-Broker Protocol Version or Kafka Metadata Version property is still set, you must clear it manually in Cloudera Manager.

Cloudera Runtime 7.3.2.100 CHF 1

There are no behavioral changes in this release.

Cloudera Runtime 7.3.2

Cloudera Runtime 7.3.2 introduces functional adjustments, behavioral updates for Kafka, and includes all service packs and cumulative hotfixes from 7.3.1.100 through 7.3.1.706. For a comprehensive record of all functional adjustments in Cloudera Runtime 7.3.1.x, see Behavioral Changes.

Component-level custom Java home configuration removed

Previous behavior:

You could configure a component-specific Java home for Kafka Connect.

New behavior:

The component-level custom Java home configuration options are removed. Kafka Connect now uses the host-level java_home configuration. If you previously set a component-specific Java home for this service, verify the host-level java_home setting after upgrading.

High Watermark no longer advances when ISR is below MinISR

Previous behavior:

The High Watermark (HWM) advanced regardless of whether the in-sync replica (ISR) count was below min.insync.replicas. When producers used acks=1 or acks=0, messages were written to the leader and became consumable once the HWM advanced, even if the ISR had dropped below the minimum threshold. The min.insync.replicas setting only affected acks=all produce requests, blocking writes when ISR was insufficient, but did not prevent HWM advancement or consumer reads for acks=0/acks=1 messages.

New behavior:

The HWM no longer advances when the ISR count falls below min.insync.replicas. As a result, consumers are blocked from reading new messages in this condition, even if producers with acks=1 or acks=0 are still writing to the leader. This ensures data is only consumable when it meets the cluster's minimum durability requirements. If you use min.insync.replicas=2 or higher, you may see reduced consumer throughput when the ISR count drops below the configured minimum.

For more information, see KIP-966: Eligible Leader Replicas and KAFKA-15583.

Kafka protocol version is set automatically during upgrades

Previous behavior:

The inter.broker.protocol.version property for ZooKeeper-based clusters and the metadata.version property for KRaft-based clusters were not set automatically before an upgrade. Manually configuring these properties to the current protocol and metadata version was required before an upgrade.

New behavior:

During a cluster upgrade, Cloudera Manager now automatically sets the inter.broker.protocol.version property for ZooKeeper-based clusters and the metadata.version property for KRaft-based clusters. Manual configuration is no longer required.