Flume

This release provides Flume 1.5.2 with no additional Apache patches.

• FLUME-2050: Upgrade to Log4j 2.10.0.

• FLUME-2678: Refactor Hive dependencies..

• FLUME-2973: Deadlock in hdfs sink.

• FLUME-3132: Upgrade tomcat jasper library dependencies.

• FLUME-3194: upgrade derby to the latest version.

HDP 2.6.4 provided Flume 1.5.2 with no additional Apache patches.

HDP 2.6.3 provided Flume 1.5.2 with no additional Apache patches.

HDP 2.6.2 provided Flume 1.5.2 and the following Apache patches:

• FLUME-2427: Be less verbose in isFileClosed() notification.

• FLUME-2672: NPE in KafkaSourceCounter.

• FLUME-2821: Flume-Kafka Source with new Consumer.

• FLUME-2822: Flume-Kafka-Sink with new Producer.

• FLUME-2852: Kafka Source/Sink should optionally read/write Flume records.

• FLUME-2915: The kafka channel using new APIs will be stuck when the sink is avro sink.

• FLUME-2920: Kafka Channel Should Not Commit Offsets When Stopping.

• FLUME-2954: Make raw data appearing in log messages explicit.

• FLUME-2972: Handle offset migration in the new Kafka Channel.

• FLUME-2983: Handle offset migration in the new Kafka Source.

• FLUME-3114: Upgrade commons-httpclient library dependency.

HDP 2.6.1 provided Flume 1.5.2 with no additional Apache patch.

HDP 2.6.0 provided Flume 1.5.2 and the following Apache patch:

• FLUME-1334: Flume startup script for Windows.

• FLUME-1734: Hive Sink based on the new Hive Streaming support.

• FLUME-2058: TestFlumeEventQueue in FileChannel fails on Windows.

• FLUME-2068: File Channel issue - recovering from BadCheckpoint exception on Windows.

• FLUME-2095: JMS source with TIBCO (patch-1).

• FLUME-2122: Minor cleanups of User guide.

• FLUME-2123: Morphline Solr sink missing short type name.

• FLUME-2136: Fix intermittent test failure in TestMonitoredCounterGroup on Windows.

• FLUME-2137: Fix StagedInstall.java to invoke the correct startup script on Windows.

• FLUME-2145: TestCheckpointRebuilder.testFastReplay fails on Windows.

• FLUME-2146: Windows: Tmp file creation in TestBodyTextEventSerializer.java needs fixing.

• FLUME-2150: Fix TestFileChannelEncryption failure on Windows.

• FLUME-2151: Windows: Update TestExecSource to use native commands on Windows.

• FLUME-2162: TestHDFSEventSinkOnMiniCluster.maxUnderReplicationTest fails on hadoop2.

• FLUME-2175: Update Developer Guide with notes on how to upgrade Protocol Buffer version.

• FLUME-2218: TestFileChannelIntegrityTool tests failing on Windows.

• FLUME-2219: Windows: Flume tests need to know location of Hadoop native libraries (hadoop.dll).

• FLUME-2224: Disable File channel dual checkpointing on Windows.

• FLUME-2226: Refactor BlobHandler out of morphline sink and into HTTP source.

• FLUME-2227: Move BlobDeserializer from Morphline Sink to flume-ng-core.

• FLUME-2337: export JAVA_HOME in flume-env.sh.template and increase heap size.

• FLUME-2358: File Channel needs to close BackingStore & EventQueue before deleting files in checkpoint directory.

• FLUME-2359: TestFileChannelIntegrityTool throws exception on class teardown on Windows.

• FLUME-2402: Warning seen when overflow is disabled for Spillable Channel.

• FLUME-2407: Spillable Channel sometimes fails on reconfigure.

• FLUME-2412: Improve Logging in Spillable Channel.

• FLUME-2441: Unit test TestHTTPSource.java failed with IBM JDK 1.7.

• FLUME-2442: Need an alternative to providing clear text passwords in flume config.

• FLUME-2450: Improve replay index insertion speed.

• FLUME-2451: HDFS Sink Cannot Reconnect After NameNode Restart.

• FLUME-2501: Updating HttpClient lib version to ensure compat with Solr.

• FLUME-2508: LineDeserializer causes incorrect behavior in SpoolDir Source on Windows.

• FLUME-2511: Allow configuration of enabled protocols in Avro source and Rpc client.

• FLUME-2520: HTTP Source should be able to block a prefixed set of protocols..

• FLUME-2530: Resource leaks found by Coverity tool.

• FLUME-2532: Windows : TestReliableSpoolingFileEventReader - need to close filereader.

• FLUME-2533: HTTPS tests fail on Java 6.

• FLUME-2534: Windows : Windows : TestFlumeEventQueue has many failures.

• FLUME-2541: Bug in TestBucketWriter.testSequenceFileCloseRetries.

• FLUME-2586: HDFS Sink should have an option to try rename even if close fails.

• FLUME-2595: Add option to checkpoint on file channel shutdown.

• FLUME-2624: Streaming ingest performance improvement.

• FLUME-2632: High CPU on KafkaSink.

• FLUME-2655: Update documentation for hdfs.closeTries based on FLUME-2586.

• FLUME-2662: Upgrade to Commons-IO 2.4.

• FLUME-2663: Address Build warnings of duplicate dependencies listed.

• FLUME-2722: Windows : TestKafkaSourceUtil Unit Tests fail on Windows.

• FLUME-2729: Allow pollableSource backoff times to be configurable.

• FLUME-2754: Hive Sink skipping first transaction in each Batch of Hive Transactions.

• FLUME-2761: Move Hive sink out of preview mode.

• FLUME-2804: Hive sink should try to clean up transactions when flume exits.

• FLUME-2812: Fix semaphore leak causing java.lang.Error: Maximum permit count exceeded in MemoryChannel.

• FLUME-2841: Upgrade commons-collections to 3.2.2.

• FLUME-2854: parameterizing jetty version..

• FLUME-2865: Upgrade thrift version (0.9.2).

• FLUME-2947: Upgrade Hive and thrift dependencies.

• FLUME-2991: ExecSource command execution starts before starting the sourceCounter.

• FLUME-3031: Sequence source should reset its counter for event body on channel exception.

• FULME-2544: Windows: Incorrect Path Separator used in HDFS path (HDFS Sink).