Cloudera Distribution of Apache Kafka Known Issues

Flume Kafka Client Incompatible Changes in CDH 5.8

After an upgrade to CDH 5.8, Flume throws an error on startup and stops processing Kafka messages. This is caused by an incompatible configuration change: upgrade of the Kafka client from 0.8.2 to 0.9.0 in CDH 5.8.

Additionally, Flume may skip some Kafka messages because Kafka offset storage moves from ZooKeeper to Kafka with the client change. Without knowing about the previous offsets stored in ZooKeeper, the Flume Kafka client assumes the most recent offset, skipping over the existing messages that have not yet been read.

Products affected: Flume

Releases affected: CDH 5.8.0, CDH 5.8.1

Users affected: Users who have Flume configured with a Kafka source or channel. Flume Kafka sinks are not affected.

Severity (Low/Medium/High): High

Impact: Potential loss of messages consumed by the Flume agent from Kafka source, or loss of events in the Flume Kafka channel after upgrade

Immediate action required: If you are upgrading from a version lower than CDH 5.7.0, upgrade to CDH 5.7.x first. The Flume Kafka client version in lower versions of CDH does not support the migration properties required in the Flume configuration.

If you are currently running CDH 5.7.x, upgrade to CDH 5.8.2 or CDH 5.9.0 to resolve this issue. If you are upgrading to CDH 5.8.0 or CDH 5.8.1, see the following documentation to perform the steps required for Flume before upgrading: http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_58_upgrade_kafka_flume_note.html

Addressed in release/refresh/patch: CDH 5.8.2 and higher, CDH 5.9.0 and higher

Cloudera Bug: TSB-173

Workaround: See Upgrading to CDH 5.8 When Using the Flume Kafka Client

Cloudera Manager 5.12.1 Installs the Latest Version of CDK powered By Apache Kafka by Default

Installing Kafka through Cloudera Manager 5.12.1 installs the latest available version of CDK Powered By Apache Kafka by default.

Workaround: In Cloudera Manager, go to Hosts > Parcels > Configuration and change the appropriate remote parcel repository to a parcel repository that hosts the required version of CDK Powered by Apache Kafka. Once the parcel repository is set, install the correct version of CDK Powered by Apache Kafka from the parcel. For more information regarding parcel configuration and installation, see Parcel Configuration Settings, and Installing or Upgrading Kafka from a Parcel.

Affected Versions: CDH 5.12.1

Fixed Versions: 5.13.0

Cloudera Issue: OPSAPS-43222