Cloudera Docs

What's New in Apache Kafka

Learn about the new features of Apache Kafka in Cloudera Runtime 7.3.1, its service packs and cumulative hotfixes.

Cloudera Runtime 7.3.1.500 SP3:

The LdapLoingModule is blocked by default
The LdapLoginModule (com.sun.security.auth.module.LdapLoginModule) class is blocked by default and can not be used in SASL JAAS configurations. Use the PlainLoginModule (org.apache.kafka.common.security.plain.PlainLoginModule) class for LDAP authentication.
Configurable allow list of URLs for OAuth authentication
A new Java option, org.apache.kafka.sasl.oauthbearer.allowed.urls, is introduced for Kafka brokers and Kafka Connect that enable you to specify an allow list of URLs for the OAuth authentication mechanism. This gives you stricter control over which URLs can be accessed by Kafka for authentication.

By default the option is empty, which means that a connection to any URL is allowed. The configuration property you use in Cloudera Manager to set the option is different for Kafka brokers and Kafka Connect. Use the following configuration properties for Kafka brokers and Kafka Connect:

  • For Kafka brokers, configure the option by adding it to the Additional Broker Java Options Kafka broker property in Cloudera Manager. For example:
    -Dorg.apache.kafka.sasl.oauthbearer.allowed.urls=http://www.oauth-example-1.com,http://www.oauth-example-2.com
  • For Kafka Connect, configure the option by adding it to the EXTRA_ARGS environment variable. The EXTRA_ARGS environment variable is configured by adding it to the Kafka Connect Environment Advanced Configuration Snippet (Safety Valve) property in Cloudera Manager. For example: 
    EXTRA_ARGS=-Dorg.apache.kafka.sasl.oauthbearer.allowed.urls="http://www.oauth-example-1.com,http://www.oauth-example-2.com"

Cloudera Runtime 7.3.1.400 SP2:

There are no new features in this release.

Cloudera Runtime 7.3.1.300 SP1 CHF 1:

There are no new features in this release.

Cloudera Runtime 7.3.1.200 SP1:

There are no new features in this release.

Cloudera Runtime 7.3.1.100 CHF 1:

There are no new features in this release.

Cloudera Runtime 7.3.1:

Kafka Rolling Restart check—all partitions fully replicated
A new broker rolling restart check option, all partitions fully replicated has been introduced. Selecting this option ensures that all partitions are in a fully synchronized state when a broker is stopped. For more information, see Rolling restart checks.
Kafka is safely stopped during operating system upgrades
During OS upgrades, Cloudera Manager now ensures that Kafka brokers are safely stopped. Specifically, Cloudera Manager now performs a rolling restart check before stopping a broker. This ensures that the Kafka service stays healthy during the upgrade. The level of health guarantee that Cloudera Manager ensures is determined by the restart check type set in the Cluster Health Guarantee During Rolling Restart Kafka property. Cloudera recommends that you set this property to all partitions stay healthy to avoid service outages. For more information, see Rolling restart checks.
useSubjectCredsOnly set to true by default in Kafka Connect

In previous versions, the javax.security.auth.useSubjectCredsOnly JVM property was set to false in Kafka Connect. Because of this, connectors running with an invalid or no JAAS configuration could use the credentials of other connectors to establish connections. Starting with this release, useSubjectCredsOnly is set to true by default. As a result, connectors are required to use their own credentials.

This default change is true for newly provisioned clusters. On upgraded clusters, useSubjectCredsOnly remains set to false to ensure backwards compatibility. If you are migrating connectors from a cluster running a previous version of Cloudera Runtime to a new cluster running 7.2.18 or later, you must ensure that credentials are added to the connector configuration when migrated. Otherwise, migrated connectors may not work on the new cluster.

In addition to the default value change, a new Kafka Connect property is introduced in Cloudera Manager that you can use to set useSubjectCredsOnly. The property is called Add Use Subject Credentials Only JVM Option With True Value. Setting this property to false does not expressly set useSubjectCredsOnlyto false. Instead, it sets useSubjectCredsOnly to the cluster default value.

Persistent MQTT sessions support for the MQTT Source connector

Version 1.1.0 of the MQTT Source connector is released. The connector now supports MQTT persistent sessions. This enables the connector to resume (persist) a previous session with an MQTT broker after a session is interrupted. Enabling this feature can ensure that no messages are lost if the connector is momentarily stopped or if the network connection is interrupted.

To support persistent sessions, the following new properties are introduced:

  • MQTT Client ID

    This property specifies the MQTT client ID that the connector uses.

  • MQTT Clean Session

    This property controls whether the connector should start clean or persistent sessions. Set this property to false to enable persistent sessions.

Existing connectors will continue to function, upgrading them, however, is not possible. If you want to use the new version of the connector, you must deploy a new instance of the connector. For more information, see MQTT Source connector and MQTT Source properties reference.