HDP-2.3.4 Release Notes
Also available as:
PDF

Kafka

[Important]Important

Hortonworks strongly recommends that all users running HDP 2.3.4 upgrade to HDP 2.3.4.7.

HDP 2.3.4 provides Kafka 0.9.0 and the following Apache patches:

  • KAFKA-2803: Add hard bounce system test for Kafka Connect.

  • KAFKA-2812: improve consumer integration tests.

  • KAFKA-2862: Fix MirrorMaker's message.handler.args description.

  • KAFKA-2872: unite sink nodes with parent nodes in addSink.

  • KAFKA-2877: handle request timeout in sync group.

  • KAFKA-2878: Guard against OutOfMemory in Kafka broker.

  • KAFKA-2879: Make MiniKDC test service slightly more generic.

  • KAFKA-2880: consumer should handle disconnect/timeout for metadata requests.

  • KAFKA-2881: Improve Consumer Configs and API Documentation.

  • KAFKA-2882: Add constructor cache for Snappy and LZ4 Output/Input streams in Compressor.java.

  • KAFKA-2892: Consumer Docs Use Wrong Method.

  • KAFKA-2899: improve logging when unexpected exceptions thrown in reading local log.

  • KAFKA-2906: Fix Connect javadocs, restrict only to api subproject, and clean up javadoc warnings.

  • KAFKA-2913: missing partition check when removing groups from cache.

  • KAFKA-2942: inadvertent auto-commit when pre-fetching can cause message loss.

  • KAFKA-2950: Fix performance regression in the producer.

HDP 2.3.2 provided Kafka 0.8.2, with no additional Apache patches.

HDP 2.3.0 provided Kafka 0.8.2 and the following Apache patches:

  • KAFKA-1005 Shutdown consumer at the end of consumer performance test.

  • KAFKA-1416 Unify sendMessages in TestUtils

  • KAFKA-1461 Replica fetcher thread does not implement any back-off behavior

  • KAFKA-1499 trivial follow-up (remove unnecessary parentheses)

  • KAFKA-1501 Let the OS choose the port in unit tests to avoid collisions

  • KAFKA-1517 Messages is a required argument to Producer Performance Test

  • KAFKA-1546 Automate replica lag tuning;

  • KAFKA-1634 Bump up Offset Commit Request to v2 to add global retention and remove per-partition commit timestamp

  • KAFKA-1664 Kafka does not properly parse multiple ZK nodes with non-root chroot

  • KAFKA-1683 add Session concept in SocketServer.PlainTextTransportLayer fixes.

  • KAFKA-1684 Kerberos/SASL implementation.

  • KAFKA-1688 Adding all public entities for adding a pluggable authorizer to Kafka.

  • KAFKA-1755 Reject compressed and unkeyed messages sent to compacted topics

  • KAFKA-1809 Refactor brokers to allow listening on multiple ports and IPs

  • KAFKA-1824 ConsoleProducer - properties key.separator and parse.key no longer work

  • KAFKA-1845 KafkaConfig should use ConfigDef

  • KAFKA-1852 Reject offset commits to unknown topics

  • KAFKA-1863 Add docs for possible thrown exception in Callback;

  • KAFKA-1865 Add a flush() method to the producer.

  • KAFKA-1866 LogStartOffset gauge throws exceptions after log.delete()

  • KAFKA-1910 Refactor new consumer and fixed a bunch of corner cases / unit tests

  • KAFKA-1926 Replace kafka.utils.Utils with o.a.k.common.utils.Utils

  • KAFKA-1961 Prevent deletion of _consumer_offsets topic

  • KAFKA-1973 Remove the accidentally created LogCleanerManager.scala.orig

  • KAFKA-1982 (add missing files) change kafka.examples.Producer to use the new java producer

  • KAFKA-1982 change kafka.examples.Producer to use the new java producer

  • KAFKA-1986 Request failure rate should not include invalid message size and offset out of range

  • KAFKA-1988 Fix org.apache.kafka.common.utils.Utils.abs and add Partitioner.toPositive

  • KAFKA-1989 New purgatory design; patched by Yasuhiro Matsuda

  • KAFKA-1990 Add unlimited time-based log retention

  • KAFKA-1992 checkEnoughReplicasReachOffset doesn't need to get requiredAcks

  • KAFKA-1994 Evaluate performance effect of chroot check on Topic creation

  • KAFKA-1996 Fix scaladoc error.

  • KAFKA-1997 Refactor MirrorMaker based on KIP-3;

  • KAFKA-2002 Mx4JLoader doesn't disable when kafka_mx4jenable=false.

  • KAFKA-2009 Fix two minor bugs in mirror maker.

  • KAFKA-2013 benchmark test for the purgatory

  • KAFKA-2016 RollingBounceTest takes long

  • KAFKA-2024 Log compaction can generate unindexable segments.

  • KAFKA-2033 Small typo in documentation

  • KAFKA-2034 sourceCompatibility not set in Kafka build.gradle

  • KAFKA-2039 Update Scala to 2.10.5 and 2.11.6

  • KAFKA-2042 Update topic list of the metadata regardless of cluster information;

  • KAFKA-2043 CompressionType is passed in each RecordAccumulator append

  • KAFKA-2044 Support requests and responses from o.a.k.common in KafkaApis

  • KAFKA-2047 Move the stream creation into concurrent mirror maker threads

  • KAFKA-2048 Change lock synchronized to inLock() for partitionMapCond

  • KAFKA-2050 Avoid calling .size() on linked list.

  • KAFKA-2056 Fix transient testRangePartitionAssignor failure

  • KAFKA-2088 kafka-console-consumer.sh should not create zookeeper path when no brokers found and chroot was set in zookeeper.connect.

  • KAFKA-2090 Remove duplicate check to metadataFetchInProgress

  • KAFKA-2096 Enable keepalive socket option for broker to prevent socket leak

  • KAFKA-2099 BrokerEndPoint file, methods and object names should match

  • KAFKA-2104 testDuplicateListeners() has a typo

  • KAFKA-2109 Support retries in KafkaLog4jAppender

  • KAFKA-2112 make overflowWheel volatile

  • KAFKA-2113 TestPurgatoryPerformance does not compile using IBM JDK

  • KAFKA-2114 Unable to change min.insync.replicas default.

  • KAFKA-2115 Error updating metrics in RequestChannel

  • KAFKA-2117 Use the correct metadata field for reading offset struct

  • KAFKA-2118 Cleaner cannot clean after shutdown during replaceSegments.

  • KAFKA-2119 ConsumerRecord key() and value() methods should not have throws Exception

  • KAFKA-2121 Close internal modules upon client shutdown

  • KAFKA-2122 Remove controller.message.queue.size Config

  • KAFKA-2128 kafka.Kafka should return non-zero exit code when caught exception.

  • KAFKA-2131 Update new producer javadocs with correct documentation links

  • KAFKA-2138 Fix producer to honor retry backoff

  • KAFKA-2140 Improve code readability

  • KAFKA-527 Compression support does numerous byte copies;