DocsHortonworks Data Platform

 

 5.4. Flume

HDP 2.4.0 provides Flume 1.5.2 and the following Apache patches:

HDP 2.3.4 provided Flume 1.5.2 and the following Apache patches:

HDP 2.3.2 provided Flume 1.5.2, with no additional Apache patches.

HDP 2.3.0 provided Flume 1.5.2 and the following Apache patches:

NEW FEATURES

  • FLUME-1734 Hive Sink based on the new Hive Streaming support.

  • FLUME-2442 Need an alternative to providing clear text passwords in flume config.

Kafka Sink (preview)

  • FLUME-2251 Add support for Kafka Sink.

  • FLUME-2454 Support batchSize to allow multiple events per transaction to the Kafka Sink.

  • FLUME-2455 Documentation update for Kafka Sink.

  • FLUME-2470 Kafka Sink and Source must use camel case for all configs.

  • FLUME-2499 Include Kafka Message Key in Event Header, Updated Comments.

Kafka Source

IMPROVEMENTS

  • FLUME-2095 JMS source with TIBCO (patch-1).

  • FLUME-2226 Refactor BlobHandler out of morphline sink and into HTTP source.

  • FLUME-2227 Move BlobDeserializer from Morphline Sink to flume-ng-core.

  • FLUME-2337 export JAVA_HOME in flume-env.sh.template and increase heap size.

  • FLUME-2450 Improve replay index insertion speed.

  • FLUME-2511 Allow configuration of enabled protocols in Avro source and Rpc client.

  • FLUME-2586 HDFS Sink should have an option to try rename even if close fails.

  • FLUME-2595 Add option to checkpoint on file channel shutdown.

  • FLUME-2624 Streaming ingest performance improvement.

  • FLUME-2662 Upgrade to Commons-IO 2.4.

  • FLUME-2663 Address Build warnings of duplicate dependencies listed.

  • FLUME-2665 Update documentation for hdfs.closeTries based on FLUME-2586.

BUG FIXES

  • FLUME-2122 Minor cleanups of User guide.

  • FLUME-2123 Morphline Solr sink missing short type name.

  • FLUME-2162 TestHDFSEventSinkOnMiniCluster.maxUnderReplicationTest fails on hadoop2.

  • FLUME-2175 Update Developer Guide with notes on how to upgrade Protocol Buffer version.

  • FLUME-2358 File Channel needs to close BackingStore and EventQueue before deleting files in checkpoint directory.

  • FLUME-2402 Warning seen when overflow is disabled for Spillable Channel.

  • FLUME-2407 Spillable Channel sometimes fails on reconfigure.

  • FLUME-2412 Improve Logging in Spillable Channel.

  • FLUME-2441 Unit test TestHTTPSource.java failed with IBM JDK 1.7.

  • FLUME-2451 HDFS Sink Cannot Reconnect After NameNode Restart.

  • FLUME-2501 Updating HttpClient lib version to ensure compat with Solr.

  • FLUME-2520 HTTP Source should be able to block a prefixed set of protocols.

  • FLUME-2530 Resource leaks found by Coverity tool.

  • FLUME-2533 HTTPS tests fail on Java 6.

  • FLUME-2541 Bug in TestBucketWriter.testSequenceFileCloseRetries.

Legal notices