Kafka
In addition to any Apache patches ported over from the 2.4.x core, HDP 2.3.6 provides Kafka 0.9.0.1 and the following Apache patches:
KAFKA-3448: Support zone index in IPv6 regex.
HDP 2.3.6 provides Kafka 0.9.0.1 and ports the following Apache patches from the 2.4.x core:
KAFKA-2803: Add hard bounce system test for Kafka Connect.
KAFKA-2812: Improve consumer integration tests.
KAFKA-2862: Fix MirrorMaker's message.handler.args description.
KAFKA-2872: Unite sink nodes with parent nodes in addSink.
KAFKA-2877: Handle request timeout in sync group.
KAFKA-2878: Guard against OutOfMemory in Kafka broker.
KAFKA-2879: Make MiniKDC test service slightly more generic.
KAFKA-2880: Consumer should handle disconnect/timeout for metadata requests.
KAFKA-2881: Improve Consumer Configs and API Documentation.
KAFKA-2882: Add constructor cache for Snappy and LZ4 Output/Input streams in Compressor.java.
KAFKA-2892: Consumer Docs Use Wrong Method.
KAFKA-2899: Improve logging when unexpected exceptions thrown in reading local log.
KAFKA-2906: Fix Connect javadocs, restrict only to api subproject, and clean up javadoc warnings.
KAFKA-2913: Missing partition check when removing groups from cache.
KAFKA-2942: Inadvertent auto-commit when pre-fetching can cause message loss.
KAFKA-2950: Fix performance regression in the producer.
HDP 2.3.4.7 provided Kafka 0.9.0 with no additional Apache patches.
HDP 2.3.4 provided Kafka 0.9.0 and the following Apache patches:
KAFKA-2803: Add hard bounce system test for Kafka Connect.
KAFKA-2812: improve consumer integration tests.
KAFKA-2862: Fix MirrorMaker's message.handler.args description.
KAFKA-2872: unite sink nodes with parent nodes in addSink.
KAFKA-2877: handle request timeout in sync group.
KAFKA-2878: Guard against OutOfMemory in Kafka broker.
KAFKA-2879: Make MiniKDC test service slightly more generic.
KAFKA-2880: consumer should handle disconnect/timeout for metadata requests.
KAFKA-2881: Improve Consumer Configs and API Documentation.
KAFKA-2882: Add constructor cache for Snappy and LZ4 Output/Input streams in Compressor.java.
KAFKA-2892: Consumer Docs Use Wrong Method.
KAFKA-2899: improve logging when unexpected exceptions thrown in reading local log.
KAFKA-2906: Fix Connect javadocs, restrict only to api subproject, and clean up javadoc warnings.
KAFKA-2913: missing partition check when removing groups from cache.
KAFKA-2942: inadvertent auto-commit when pre-fetching can cause message loss.
KAFKA-2950: Fix performance regression in the producer.
HDP 2.3.2 provided Kafka 0.8.2, with no additional Apache patches.
HDP 2.3.0 provided Kafka 0.8.2 and the following Apache patches:
KAFKA-1005 Shutdown consumer at the end of consumer performance test.
KAFKA-1416 Unify sendMessages in TestUtils
KAFKA-1461 Replica fetcher thread does not implement any back-off behavior
KAFKA-1499 trivial follow-up (remove unnecessary parentheses)
KAFKA-1501 Let the OS choose the port in unit tests to avoid collisions
KAFKA-1517 Messages is a required argument to Producer Performance Test
KAFKA-1546 Automate replica lag tuning;
KAFKA-1634 Bump up Offset Commit Request to v2 to add global retention and remove per-partition commit timestamp
KAFKA-1664 Kafka does not properly parse multiple ZK nodes with non-root chroot
KAFKA-1683 add Session concept in SocketServer.PlainTextTransportLayer fixes.
KAFKA-1684 Kerberos/SASL implementation.
KAFKA-1688 Adding all public entities for adding a pluggable authorizer to Kafka.
KAFKA-1755 Reject compressed and unkeyed messages sent to compacted topics
KAFKA-1809 Refactor brokers to allow listening on multiple ports and IPs
KAFKA-1824 ConsoleProducer - properties key.separator and parse.key no longer work
KAFKA-1845 KafkaConfig should use ConfigDef
KAFKA-1852 Reject offset commits to unknown topics
KAFKA-1863 Add docs for possible thrown exception in Callback;
KAFKA-1865 Add a flush() method to the producer.
KAFKA-1866 LogStartOffset gauge throws exceptions after log.delete()
KAFKA-1910 Refactor new consumer and fixed a bunch of corner cases / unit tests
KAFKA-1926 Replace kafka.utils.Utils with o.a.k.common.utils.Utils
KAFKA-1961 Prevent deletion of _consumer_offsets topic
KAFKA-1973 Remove the accidentally created LogCleanerManager.scala.orig
KAFKA-1982 (add missing files) change kafka.examples.Producer to use the new java producer
KAFKA-1982 change kafka.examples.Producer to use the new java producer
KAFKA-1986 Request failure rate should not include invalid message size and offset out of range
KAFKA-1988 Fix org.apache.kafka.common.utils.Utils.abs and add Partitioner.toPositive
KAFKA-1989 New purgatory design; patched by Yasuhiro Matsuda
KAFKA-1990 Add unlimited time-based log retention
KAFKA-1992 checkEnoughReplicasReachOffset doesn't need to get requiredAcks
KAFKA-1994 Evaluate performance effect of chroot check on Topic creation
KAFKA-1996 Fix scaladoc error.
KAFKA-1997 Refactor MirrorMaker based on KIP-3;
KAFKA-2002 Mx4JLoader doesn't disable when kafka_mx4jenable=false.
KAFKA-2009 Fix two minor bugs in mirror maker.
KAFKA-2013 benchmark test for the purgatory
KAFKA-2016 RollingBounceTest takes long
KAFKA-2024 Log compaction can generate unindexable segments.
KAFKA-2033 Small typo in documentation
KAFKA-2034 sourceCompatibility not set in Kafka build.gradle
KAFKA-2039 Update Scala to 2.10.5 and 2.11.6
KAFKA-2042 Update topic list of the metadata regardless of cluster information;
KAFKA-2043 CompressionType is passed in each RecordAccumulator append
KAFKA-2044 Support requests and responses from o.a.k.common in KafkaApis
KAFKA-2047 Move the stream creation into concurrent mirror maker threads
KAFKA-2048 Change lock synchronized to inLock() for partitionMapCond
KAFKA-2050 Avoid calling .size() on linked list.
KAFKA-2056 Fix transient testRangePartitionAssignor failure
KAFKA-2088 kafka-console-consumer.sh should not create ZooKeeper path when no brokers found and chroot was set in zookeeper.connect.
KAFKA-2090 Remove duplicate check to metadataFetchInProgress
KAFKA-2096 Enable keepalive socket option for broker to prevent socket leak
KAFKA-2099 BrokerEndPoint file, methods and object names should match
KAFKA-2104 testDuplicateListeners() has a typo
KAFKA-2109 Support retries in KafkaLog4jAppender
KAFKA-2112 make overflowWheel volatile
KAFKA-2113 TestPurgatoryPerformance does not compile using IBM JDK
KAFKA-2114 Unable to change min.insync.replicas default.
KAFKA-2115 Error updating metrics in RequestChannel
KAFKA-2117 Use the correct metadata field for reading offset struct
KAFKA-2118 Cleaner cannot clean after shutdown during replaceSegments.
KAFKA-2119 ConsumerRecord key() and value() methods should not have throws Exception
KAFKA-2121 Close internal modules upon client shutdown
KAFKA-2122 Remove controller.message.queue.size Config
KAFKA-2128 kafka.Kafka should return non-zero exit code when caught exception.
KAFKA-2131 Update new producer javadocs with correct documentation links
KAFKA-2138 Fix producer to honor retry backoff
KAFKA-2140 Improve code readability
KAFKA-527 Compression support does numerous byte copies;