What's New in Apache Kafka
Learn about the new features of Apache Kafka in Cloudera Runtime 7.2.17.
Rebase on Kafka 3.4.0
Kafka shipped with this version of Cloudera Runtime is based on Apache Kafka 3.4.0. For more information, see the following upstream resources:
Kafka KRaft [TECHNICAL PREVIEW]
Apache Kafka Raft (KRaft) is a consensus protocol used for metadata management that was developed as a replacement for Apache ZooKeeper. Using KRaft for managing Kafka metadata instead of ZooKeeper offers various benefits including a simplified architecture and a reduced operational footprint.
- Deployments with multiple log directories. This includes deployments that use JBOD for storage.
- Delegation token based authentication.
- Migrating an already running Kafka service from ZooKeeper to KRaft.
- Atlas Integration.
For a conceptual overview on KRaft, see Kafka KRaft. For more information on how to deploy a Streams Messaging Data Hub cluster that is running KRaft mode, see Setting up your Streams Messaging cluster.
SMT plugins for binary conversion
Two Cloudera developed Single Message Transforms (SMT) plugins are added. These
ConvertFromBytes plugins, which
you can use to convert binary data to or from the Kafka Connect internal data format.
EOS for source connectors
Exactly-once semantics (EOS) support is added for Kafka Connect source connectors. For more information, see Configuring EOS for source connectors.
Rolling restart checks provide a high cluster health guarantees by default
The default value of the Cluster Health Guarantee During Rolling
Restart property is changed from
partitions stay healthy. This property defines what type of checks are performed
during a Rolling Restart on the restarted broker. Each setting guarantees a different level
of cluster health during Rolling Restarts. With the
none setting, no checks
are performed. This means that in previous versions no guarantees were provided on cluster
health by default.
The new default,
healthy partitions stay healthy, ensures a high
level of guarantees on cluster health. This setting ensures that no partitions go into an
under-min-isr state when a broker is stopped. This is achieved by waiting before each broker
is stopped so that all other brokers can catch up with all replicas that are in an
at-min-isr state. Additionally, the setting ensures that the restarted broker is accepting
requests on its service port before restarting the next broker. This setting ignores
partitions which are already in an under-min-isr state. For more information, see Configuring EOS for source connectors.
LDAPS SSL configurations are inherited from the Kafka broker
The SSL configurations of LDAP over SSL (LDAPS) are inherited from the Kafka broker. Previously, the JDK default was used. If the JDK default certificate store contains certificates which were used to setup SSL connection to LDAP, it should be imported to the broker stores.
Aliases for Kafka CLI tools
kafka-features.sh command line
tools. These tools can now be called globally with