What's new in Apache Kafka

This topic lists new features for Apache Kafka in this release of Cloudera Data Flow for Data Hub.

Rebase on Apache Kafka 2.3.0

Kafka available with this version of Cloudera Data Flow for Data Hub is based on Apache Kafka 2.3.0. For more information, see Apache Kafka Notable Changes and Apache Kafka Release Notes in the upstream documentation.

Access to Kafka Metadata in Zookeeper is restricted by default

The Enable Zookeeper ACL (zookeeper.set.acl) property is now directly configurable in Cloudera Manager and is enabled by default. As a result of this change, access to Kafka metadata stored in Zookeeper is restricted by default. The data is still world readable, however, administrative operations, for example topic creation, deletion, any configuration changes and so on, can only be performed by authorized users. For more information, see Restrict access to Kafka metadata in Zookeeper and Unlock Kafka metadata in Zookeeper.

Ranger authorization support

Ranger support for Kafka is added. You can now use Ranger to provide authorization for Kafka. For more information, see Using Ranger to Provide Authorization in CDP as well as the documentation on Kafka Authorization with Ranger.

The resource-based Ranger service used by Kafka is user configurable

The resource-based Ranger service used by Kafka for authorization can now be manually configured with the Ranger service name for this Kafka cluster property in Cloudera Manager. In addition, if a resource-based service is set in Kafka that does not yet exist in Ranger, it will be automatically created after the Kafka service is restarted. The name of the newly created service is based on the value of the Ranger service name for this Kafka cluster property. For more information, see Configure the resource-based Ranger service used for authorization.

PAM authentication support

You can now configure Kafka to authenticate clients using PAM. For more information, see PAM Authentication.

LDAP authentication support

You can now configure Kafka to authenticate clients using LDAP. For more information, see LDAP Authentication.

New metric for monitoring garbage collector runs

A new metric called kafka_jvm_gc_runs is added to the Kafka service. This metric enables users to monitor the number of garbage collector runs performed on each broker.