Flume

HDP 2.3.4.7 provides Flume 1.5.2 with the patches specified below. No new additional Apache patches have been included in this release.

HDP 2.3.4 provided Flume 1.5.2 and the following Apache patches:

HDP 2.3.2 provided Flume 1.5.2, with no additional Apache patches.

HDP 2.3.0 provided Flume 1.5.2 and the following Apache patches:

NEW FEATURES

FLUME-1734 Hive Sink based on the new Hive Streaming support
FLUME-2442 Need an alternative to providing clear text passwords in flume config

Kafka Sink (preview)

FLUME-2251 Add support for Kafka Sink
FLUME-2454 Support batchSize to allow multiple events per transaction to the Kafka Sink
FLUME-2455 Documentation update for Kafka Sink
FLUME-2470 Kafka Sink and Source must use camel case for all configs.
FLUME-2499 Include Kafka Message Key in Event Header, Updated Comments

Kafka Source

IMPROVEMENTS

FLUME-2095 JMS source with TIBCO (patch-1)
FLUME-2226 Refactor BlobHandler out of morphline sink and into HTTP source
FLUME-2227 Move BlobDeserializer from Morphline Sink to flume-ng-core
FLUME-2337 export JAVA_HOME in flume-env.sh.template and increase heap size
FLUME-2450 Improve replay index insertion speed
FLUME-2511 Allow configuration of enabled protocols in Avro source and Rpc client
FLUME-2586 HDFS Sink should have an option to try rename even if close fails
FLUME-2595 Add option to checkpoint on file channel shutdown
FLUME-2624 Streaming ingest performance improvement
FLUME-2662 Upgrade to Commons-IO 2.4
FLUME-2663 Address Build warnings of duplicate dependencies listed
FLUME-2665 Update documentation for hdfs.closeTries based on FLUME-2586

BUG FIXES

FLUME-2122 Minor cleanups of User guide
FLUME-2123 Morphline Solr sink missing short type name
FLUME-2162 TestHDFSEventSinkOnMiniCluster.maxUnderReplicationTest fails on hadoop2
FLUME-2175 Update Developer Guide with notes on how to upgrade Protocol Buffer version
FLUME-2358 File Channel needs to close BackingStore and EventQueue before deleting files in checkpoint directory
FLUME-2402 Warning seen when overflow is disabled for Spillable Channel
FLUME-2407 Spillable Channel sometimes fails on reconfigure
FLUME-2412 Improve Logging in Spillable Channel
FLUME-2441 Unit test TestHTTPSource.java failed with IBM JDK 1.7
FLUME-2451 HDFS Sink Cannot Reconnect After NameNode Restart
FLUME-2501 Updating HttpClient lib version to ensure compatibility with Solr
FLUME-2520 HTTP Source should be able to block a prefixed set of protocols.
FLUME-2530 Resource leaks found by Coverity tool
FLUME-2533 HTTPS tests fail on Java 6
FLUME-2541 Bug in TestBucketWriter.testSequenceFileCloseRetries