A Simple Example
The following snippet shows some of the kinds of properties that can be set using the properties file. For more detailed information, see the Apache Flume 1.5.2 documentation.
agent.sources = pstream agent.channels = memoryChannel agent.channels.memoryChannel.type = memory agent.sources.pstream.channels = memoryChannel agent.sources.pstream.type = exec agent.sources.pstream.command = tail -f /etc/passwd agent.sinks = hdfsSinkagent.sinks.hdfsSink.type = hdfs agent.sinks.hdfsSink.channel = memoryChannel agent.sinks.hdfsSink.hdfs.path = hdfs://hdp/user/root/flumetest agent.sinks.hdfsSink.hdfs.fileType = SequenceFile agent.sinks.hdfsSink.hdfs.writeFormat = Text
The source here is defined as an exec source. The agent runs a given command on startup,
which streams data to stdout
, where the source gets it.
In this case, the command is a Python test script. The channel is defined as an in-memory channel and the sink is an HDFS sink.