What's New in Apache Kafka

Rebase on Apache Kafka 2.3.0

Kafka shipped with this version of Cloudera Runtime is based on Apache Kafka 2.3.0. For more information, see Apache Kafka Notable Changes and Apache Kafka Release Notes in the upstream documentation.

Provision Kafka clusters with Data Hub

The Streams Messaging Heavy Duty and Streams Messaging Light Duty cluster templates and definitions are now available in Data Hub with advanced messaging and real-time processing on streaming data using Apache Kafka, centralized schema management using Schema Registry, and management and monitoring capabilities powered by Streams Messaging Manager. For more information, see the Data Hub documentation.

Connect Kafka clients to Data Hub provisioned clusters

Connecting clients to Data Hub clusters provisioned with the Streams Messaging cluster definitions is possible. For step-by-step instructions, see Connecting Kafka clients to Data Hub provisioned clusters.

Access to Kafka Metadata in Zookeeper is restricted by default

The Enable Zookeeper ACL (zookeeper.set.acl) property is now directly configurable in Cloudera Manager and is enabled by default. As a result of this change, access to Kafka metadata stored in Zookeeper is restricted by default. The data is still world readable, however, administrative operations, for example topic creation, deletion, any configuration changes and so on, can only be performed by authorized users. For more information, see Restrict access to Kafka metadata in Zookeeper and Unlock Kafka metadata in Zookeeper.

Ranger authorization support

Ranger support for Kafka is added. You can now use Ranger to provide authorization for Kafka. For more information, see Using Ranger to Provide Authorization in CDP as well as the documentation on Kafka Authorization with Ranger.

The resource-based Ranger service used by Kafka is user configurable

The resource-based Ranger service used by Kafka for authorization can now be manually configured with the Ranger service name for this Kafka cluster property in Cloudera Manager. In addition, if a resource-based service is set in Kafka that does not yet exist in Ranger, it will be automatically created after the Kafka service is restarted. The name of the newly created service is based on the value of the Ranger service name for this Kafka cluster property. For more information, see Configure the resource-based Ranger service used for authorization.

PAM authentication support

You can now configure Kafka to authenticate clients using PAM. For more information, see PAM Authentication.

LDAP authentication support

You can now configure Kafka to authenticate clients using LDAP. For more information, see LDAP Authentication.

New metric for monitoring garbage collector runs

A new metric called kafka_jvm_gc_runs is added to the Kafka service. This metric enables users to monitor the number of garbage collector runs performed on each broker.