HDP-2.3.4.7 Release Notes
Also available as:
PDF

Kafka

HDP 2.3.4.7 provides Kafka 0.9.0 with the patches specified below. No new additional Apache patches have been included in this release.

HDP 2.3.4 provided Kafka 0.9.0 and the following Apache patches:

  • KAFKA-2803: Add hard bounce system test for Kafka Connect.

  • KAFKA-2812: improve consumer integration tests.

  • KAFKA-2862: Fix MirrorMaker's message.handler.args description.

  • KAFKA-2872: unite sink nodes with parent nodes in addSink.

  • KAFKA-2877: handle request timeout in sync group.

  • KAFKA-2878: Guard against OutOfMemory in Kafka broker.

  • KAFKA-2879: Make MiniKDC test service slightly more generic.

  • KAFKA-2880: consumer should handle disconnect/timeout for metadata requests.

  • KAFKA-2881: Improve Consumer Configs and API Documentation.

  • KAFKA-2882: Add constructor cache for Snappy and LZ4 Output/Input streams in Compressor.java.

  • KAFKA-2892: Consumer Docs Use Wrong Method.

  • KAFKA-2899: improve logging when unexpected exceptions thrown in reading local log.

  • KAFKA-2906: Fix Connect javadocs, restrict only to api subproject, and clean up javadoc warnings.

  • KAFKA-2913: missing partition check when removing groups from cache.

  • KAFKA-2942: inadvertent auto-commit when pre-fetching can cause message loss.

  • KAFKA-2950: Fix performance regression in the producer.

HDP 2.3.2 provided Kafka 0.8.2, with no additional Apache patches.

HDP 2.3.0 provided Kafka 0.8.2 and the following Apache patches:

  • KAFKA-1005 Shutdown consumer at the end of consumer performance test.

  • KAFKA-1416 Unify sendMessages in TestUtils

  • KAFKA-1461 Replica fetcher thread does not implement any back-off behavior

  • KAFKA-1499 trivial follow-up (remove unnecessary parentheses)

  • KAFKA-1501 Let the OS choose the port in unit tests to avoid collisions

  • KAFKA-1517 Messages is a required argument to Producer Performance Test

  • KAFKA-1546 Automate replica lag tuning;

  • KAFKA-1634 Bump up Offset Commit Request to v2 to add global retention and remove per-partition commit timestamp

  • KAFKA-1664 Kafka does not properly parse multiple ZK nodes with non-root chroot

  • KAFKA-1683 add Session concept in SocketServer.PlainTextTransportLayer fixes.

  • KAFKA-1684 Kerberos/SASL implementation.

  • KAFKA-1688 Adding all public entities for adding a pluggable authorizer to Kafka.

  • KAFKA-1755 Reject compressed and unkeyed messages sent to compacted topics

  • KAFKA-1809 Refactor brokers to allow listening on multiple ports and IPs

  • KAFKA-1824 ConsoleProducer - properties key.separator and parse.key no longer work

  • KAFKA-1845 KafkaConfig should use ConfigDef

  • KAFKA-1852 Reject offset commits to unknown topics

  • KAFKA-1863 Add docs for possible thrown exception in Callback;

  • KAFKA-1865 Add a flush() method to the producer.

  • KAFKA-1866 LogStartOffset gauge throws exceptions after log.delete()

  • KAFKA-1910 Refactor new consumer and fixed a bunch of corner cases / unit tests

  • KAFKA-1926 Replace kafka.utils.Utils with o.a.k.common.utils.Utils

  • KAFKA-1961 Prevent deletion of _consumer_offsets topic

  • KAFKA-1973 Remove the accidentally created LogCleanerManager.scala.orig

  • KAFKA-1982 (add missing files) change kafka.examples.Producer to use the new java producer

  • KAFKA-1982 change kafka.examples.Producer to use the new java producer

  • KAFKA-1986 Request failure rate should not include invalid message size and offset out of range

  • KAFKA-1988 Fix org.apache.kafka.common.utils.Utils.abs and add Partitioner.toPositive

  • KAFKA-1989 New purgatory design; patched by Yasuhiro Matsuda

  • KAFKA-1990 Add unlimited time-based log retention

  • KAFKA-1992 checkEnoughReplicasReachOffset doesn't need to get requiredAcks

  • KAFKA-1994 Evaluate performance effect of chroot check on Topic creation

  • KAFKA-1996 Fix scaladoc error.

  • KAFKA-1997 Refactor MirrorMaker based on KIP-3;

  • KAFKA-2002 Mx4JLoader doesn't disable when kafka_mx4jenable=false.

  • KAFKA-2009 Fix two minor bugs in mirror maker.

  • KAFKA-2013 benchmark test for the purgatory

  • KAFKA-2016 RollingBounceTest takes long

  • KAFKA-2024 Log compaction can generate unindexable segments.

  • KAFKA-2033 Small typo in documentation

  • KAFKA-2034 sourceCompatibility not set in Kafka build.gradle

  • KAFKA-2039 Update Scala to 2.10.5 and 2.11.6

  • KAFKA-2042 Update topic list of the metadata regardless of cluster information;

  • KAFKA-2043 CompressionType is passed in each RecordAccumulator append

  • KAFKA-2044 Support requests and responses from o.a.k.common in KafkaApis

  • KAFKA-2047 Move the stream creation into concurrent mirror maker threads

  • KAFKA-2048 Change lock synchronized to inLock() for partitionMapCond

  • KAFKA-2050 Avoid calling .size() on linked list.

  • KAFKA-2056 Fix transient testRangePartitionAssignor failure

  • KAFKA-2088 kafka-console-consumer.sh should not create zookeeper path when no brokers found and chroot was set in zookeeper.connect.

  • KAFKA-2090 Remove duplicate check to metadataFetchInProgress

  • KAFKA-2096 Enable keepalive socket option for broker to prevent socket leak

  • KAFKA-2099 BrokerEndPoint file, methods and object names should match

  • KAFKA-2104 testDuplicateListeners() has a typo

  • KAFKA-2109 Support retries in KafkaLog4jAppender

  • KAFKA-2112 make overflowWheel volatile

  • KAFKA-2113 TestPurgatoryPerformance does not compile using IBM JDK

  • KAFKA-2114 Unable to change min.insync.replicas default.

  • KAFKA-2115 Error updating metrics in RequestChannel

  • KAFKA-2117 Use the correct metadata field for reading offset struct

  • KAFKA-2118 Cleaner cannot clean after shutdown during replaceSegments.

  • KAFKA-2119 ConsumerRecord key() and value() methods should not have throws Exception

  • KAFKA-2121 Close internal modules upon client shutdown

  • KAFKA-2122 Remove controller.message.queue.size Config

  • KAFKA-2128 kafka.Kafka should return non-zero exit code when caught exception.

  • KAFKA-2131 Update new producer javadocs with correct documentation links

  • KAFKA-2138 Fix producer to honor retry backoff

  • KAFKA-2140 Improve code readability

  • KAFKA-527 Compression support does numerous byte copies;