What's New in Apache Kafka
Learn about the new features of Apache Kafka in Cloudera Runtime 7.3.1.
Kafka Rolling Restart check—all partitions fully replicated
A new broker rolling restart check option, all partitions fully replicated has been introduced. Selecting this option ensures that all partitions are in a fully synchronized state when a broker is stopped. For more information, see Rolling restart checks.
Kafka is safely stopped during operating system upgrades
During OS upgrades, Cloudera Manager now ensures that Kafka brokers are safely stopped. Specifically, Cloudera Manager now performs a rolling restart check before stopping a broker. This ensures that the Kafka service stays healthy during the upgrade. The level of health guarantee that Cloudera Manager ensures is determined by the restart check type set in the Cluster Health Guarantee During Rolling Restart Kafka property. Cloudera recommends that you set this property to all partitions stay healthy to avoid service outages. For more information, see Rolling restart checks.
useSubjectCredsOnly set to true by default in Kafka Connect
In previous versions, the javax.security.auth.useSubjectCredsOnly
JVM
property was set to false
in Kafka Connect. Because of this, connectors
running with an invalid or no JAAS configuration could use the credentials of other
connectors to establish connections. Starting with this release,
useSubjectCredsOnly
is set to true
by default. As a
result, connectors are required to use their own credentials.
This default change is true for newly provisioned clusters. On upgraded clusters,
useSubjectCredsOnly
remains set to false to ensure backwards
compatibility. If you are migrating connectors from a cluster running a previous version of
Runtime to a new cluster running 7.2.18 or later, you must ensure that credentials are added
to the connector configuration when migrated. Otherwise, migrated connectors may not work on
the new cluster.
In addition to the default value change, a new Kafka Connect property is introduced in
Cloudera Manager that you can use to set useSubjectCredsOnly
. The property
is called Add Use Subject Credentials Only JVM Option With True
Value. Setting this property to false does not expressly set
useSubjectCredsOnly
to false. Instead, it sets
useSubjectCredsOnly
to the cluster default value.
Persistent MQTT sessions support for the MQTT Source connector
Version 1.1.0 of the MQTT Source connector is released. The connector now supports MQTT persistent sessions. This enables the connector to resume (persist) a previous session with an MQTT broker after a session is interrupted. Enabling this feature can ensure that no messages are lost if the connector is momentarily stopped or if the network connection is interrupted.
To support persistent sessions, the following new properties are introduced:
- MQTT Client ID
This property specifies the MQTT client ID that the connector uses.
- MQTT Clean Session
This property controls whether the connector should start clean or persistent sessions. Set this property to false to enable persistent sessions.
Existing connectors will continue to function, upgrading them, however, is not possible. If you want to use the new version of the connector, you must deploy a new instance of the connector. For more information, see MQTT Source connector and MQTT Source properties reference.