To confirm that Apache Flume is working correctly, create an example configuration file. The following snippet is a sample configuration that can be set using the properties file. For more detailed information, see the “Flume User Guide.”

a1.sources = pstream 
a1.channels = memoryChannel
a1.channels.memoryChannel.type = memory 

a1.sources.pstream.channels = memoryChannel 
a1.sources.pstream.type = exec 
a1.sources.pstream.command = tail -f /etc/passwd 

a1.sinks = hdfsSink
a1.sinks.hdfsSink.type = hdfs 
a1.sinks.hdfsSink.channel = memoryChannel
a1.sinks.hdfsSink.hdfs.path = hdfs://<NN host>:8020/tmp/flumetest 
a1.sinks.hdfsSink.hdfs.fileType = SequenceFile 
a1.sinks.hdfsSink.hdfs.writeFormat = Text

The source here is defined as an exec source. The agent runs a given command on startup, which streams data to stdout, where the source gets it. The channel is defined as an in-memory channel and the sink is an HDFS sink.

Given this configuration, you can start Flume as follows:

$ bin/flume-ng agent --conf ./conf --conf-file example.conf --name a1 -Dflume.root.logger=INFO,console

	Note
	The directory specified for --conf agrument would include a shell script flume-env.sh and potentially a log4j properties file. In this example, we pass a Java option to force Flume to log to the console and we go without a custom environment script.

After validating data in hdfs://tmp/flumetest, stop Flume and resore any backup files. Copy /etc/flume/conf to the conf directory in Flume hosts.