Known Issues in Cloudera Distribution of Apache Kafka

Unsupported features

Kafka stuck with under-replicated partitions after ZooKeeper session expires

This problem might occur when your Kafka cluster includes a large number of under-replicated Kafka partitions. One or more broker logs include messages such as the following:

          [2016-01-17 03:36:00,888] INFO Partition [__samza_checkpoint_event-creation_1,3] on broker 3: Shrinking ISR for partition [__samza_checkpoint_event-creation_1,3] from 6,5 to 5 (kafka.cluster.Partition)
          [2016-01-17 03:36:00,891] INFO Partition [__samza_checkpoint_event-creation_1,3] on broker 3: Cached zkVersion [66] not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition)
        

There will also be an indication of the ZooKeeper session expiring in one or more Kafka broker logs around the same time as the previous errors:

          INFO zookeeper state changed (Expired) (org.I0Itec.zkclient.ZkClient)
        

The log is typically in /var/log/kafka on each host where a Kafka broker is running. The location is set by the property kafka.log4j.dir in Cloudera Manager. The log name is kafka-broker-hostname.log. In diagnostic bundles, the log is under logs/ hostname-ip-address/.

Affected Versions: CDK 1.4.x, 2.0.x, 2.1.x, 2.2.x Powered By Apache Kafka

Partial Fix: CDK 3.0.0 and later Powered By Apache Kafka are less likely to encounter this issue.

Workaround: To move forward after seeing this problem, restart the Kafka brokers affected. You can restart individual brokers from the Instances tab in the Kafka service page in Cloudera Manager.

To reduce the chances of this issue happening again, do what you can to make sure ZooKeeper sessions do not expire:

• Reduce the potential for long garbage collection pauses by brokers:
  • Use a better garbage collection mechanism in the JVM, such as G1GC. You can do this by adding –XX:+UseG1GC in the broker_java_opts.
  • Increase broker heap size if it is too small (broker_max_heap_size) (be careful that you don’t choose a heap size that can cause out-of-memory problems given all the services running on the node).
• Increase the ZooKeeper session timeout configuration on brokers (zookeeper.session.timeout.ms), to reduce the likelihood that sessions expire.
• Ensure ZooKeeper itself is well resourced and not overwhelmed, so it can respond. For example, it is highly recommended to locate the ZooKeeper log directory is on its own disk.

Cloudera JIRA: CDH-42514

Apache JIRA: KAFKA-2729

The Flume and Spark connectors to Kafka shipped with CDH 5.7 and higher only work with Kafka 2.0 and higher

Use Kafka 2.0 and higher to be compatible with the Flume and Spark connectors included with CDH 5.7.x.

Kafka CSD does not configure the correct listeners for the configured security.inter.broker.protocol

If you use one of the following configuration combinations, the Kafka broker will not start:

• Kerberos on, SSL off and security.inter.broker.protocol set as PLAINTEXT.
• Kerberos on, SSL on and security.inter.broker.protocol set as SSL.
• Kerberos off, SSL off and security.inter.broker.protocol set as SASL_PLAINTEXT.
• Kerberos off, SSL on and security.inter.broker.protocol set as SASL_SSL.

Workaround:

Add a configuration to the Kafka Broker Advanced Configuration Snippet (Safety Valve) for listeners on each Kafka broker. Create a configuration statement in the following format:

listeners=<PROTOCOL>://<HOST>:[<PORT> or <SSL_PORT>],<PROTOCOL>://<HOST>:[<PORT> or <SSL_PORT>]

Only new Java clients support authentication and authorization

Workaround: Migrate to the new Java producer and consumer APIs.

Requests fail when sending to a nonexistent topic with auto.create.topics.enable set to true

The first few produce requests fail when sending to a nonexistent topic with auto.create.topics.enable set to true.

Workaround: Increase the number of retries in the Producer configuration settings.

Custom Kerberos principal names must not be used for Kerberized ZooKeeper and Kafka instances

When using ZooKeeper authentication and a custom Kerberos principal, Kerberos-enabled Kafka does not start.

Workaround: None. You must disable ZooKeeper authentication for Kafka or use the default Kerberos principals for ZooKeeper and Kafka.

Performance degradation when SSL is enabled

Significant performance degradation can occur when SSL is enabled. The impact varies, depending on your CPU type and JVM version. The reduction is generally in the range 20-50%.

AdminUtils is not binary-compatible between Cloudera Distribution of Apache Kafka 1.x and 2.x

The AdminUtils APIs have changed between Cloudera Distribution of Apache Kafka 1.x and 2.x. If your application uses AdminUtils APIs, you must modify your application code to use the new APIs before you compile your application against Cloudera Distribution of Apache Kafka 2.x.

Monitoring is not supported in Cloudera Manager 5.4

If you use Cloudera Distribution of Kafka 1.2 with Cloudera Manager 5.4, you must disable monitoring.

Authenticated clients may interfere with data replication

Authenticated Kafka users may perform an action reserved for the Broker via a manually created fetch request interfering with data replication, resulting in data loss.

Products affected: CDK Powered by Apache Kafka

Releases affected: CDK 2.0.0 to 2.2.0, CDK 3.0.0

Users affected: All users

Detected by: Rajini Sivaram (rsivaram@apache.org)

Severity (Low/Medium/High):6.3 (Medium) (CVSS:3.0/AV:N/AC:L/PR:L/UI:N/S:U/C:L/I:L/A:L)

Impact:Potential data loss due to improper replication.

CVE:CVE-2018-1288

Immediate action required: Upgrade to a newer version of CDK Powered by Apache Kafka where the issue has been fixed.

Addressed in release/refresh/patch: CDK 3.1, CDH 6.0 and higher

Knowledge article: For the latest update on this issue see the corresponding Knowledge article: TSB 2018-332: Two Kafka Security Vulnerabilities: Authenticated Kafka clients may impersonate other users and and may interfere with data replication