What's New in Apache Kafka

This topic lists new features for Apache Kafka in this release of Cloudera Runtime.

Rebase on Kafka 2.4.1

Kafka shipped with this version of Cloudera Runtime is based on Apache Kafka 2.4.1. For more information, see Apache Kafka Notable Changes for versions 2.4.0 and 2.4.1, as well as the Apache Kafka Release Notes for versions 2.4.0 and 2.4.1 in the upstream documentation.

Support for Kafka Connect Added

Support for Kafka Connect is added. In CDP Kafka Connect is implemented in the form of a Kafka service role. The role is called Kafka Connect. In addition, support for Kafka Connect is also added to SMM. Users can from now on manage, monitor, and interact with Kafka Connect either through the SMM UI or SMM REST API. For more information see the Kafka Connect documentation as well as the Kafka Connect SMM documentation.

Cloudera developed HDFS and Amazon S3 Sink Connectors Available

Alongside the addition of Kafka Connect support, two Cloudera developed connectors are also added and made available for use. These are the HDFS Sink and Amazon S3 Sink connectors. For more information, see the Connector documentation.

New Command Line Tool kafka-leader-election

The kafka-preferred-replica-election.sh command line tool has been deprecated in upstream Apache Kafka 2.4.0 It has been replaced by kafka-leader-election.sh. The new tool is available for use in Runtime. In addition, an alternative for it is also provided. The alternative is kafka-leader-election. Both kafka-preferred-replica-election.sh and its alternative, kafka-preferred-replica-election are still available for use, however these are deprecated and will be removed in a future release.

kafka-sentry command line tool alternative removed

Ranger has replaced Sentry in all versions of CDP Runtime. The kafka-sentry.sh command line tool and its alternative, kafka-sentry, have therefore been deprecated and removed.

Collection of Producer Metrics is Enabled by Default

The Enable Producer Metrics (producer.metrics.enable) property is now set to true by default, as a result collection of producer metrics is now enabled by default.

Collection of Partition Level Metrics is Enabled by Default

Cloudera Manager now collects Kafka topic partition level metrics by default. This change is introduced to make the installation of Streams Messaging Manager seamless. The change however, also introduces a limitation. For more information, see the Limitations section in Kafka Known Issues.