Apache Flume Component Guide
Also available as:
PDF

Chapter 1. Feature Updates

Apache Flume version 1.5.2 includes cumulative 1.6 features and selected 1.7 features. The table below indicates the features added to Flume 1.5.2 with each release of Hortonworks Data Platform (HDP).

[Note]Note

As of HDP 2.6.0, Flume is deprecated but still supported. It will be removed and no longer supported as of HDP 3.0.0.

Table 1.1. Features by HDP Version

HDP ReleaseAdded FeaturesAdvantages
2.6.2New Kafka Source Provides support for new Kafka Consumer API, including reading multiple Kafka topics.
 New Kafka sinkProvides support for new Kafka Producer API.
 Preserve event headersPreserve event headers when using Kafka Sink and Kafka Source together. Avro Datums can be read from and written to Kafka.
 Log privacy utilityHelps determine whether or not logging potentially sensitive information is allowed.
2.5.0Kafka ChannelUses a single Kafka topic. Provides greater reliability and better performance.
 TailDir Source Greater data reliability, even with rotating file names. Can restart tailing at the point where Flume stopped, while continuing data ingest.
2.4.0Kafka SourceReads messages from a Kafka topic. Can have multiple Kafka sources running and configure them to read a unique set of partitions for the topic.
 Kafka SinkPublishes data to a Kafka topic. Supports pull-based processing from various Flume sources.
2.3.0Hive SinkNot recommended for use in production. Streams events containing delimited text or JSON data directly into a Hive table or partition. Provides a preview feature and not.