Example: Writing from Flume to HDFS

Apache Flume is a service for collecting log data. You can capture events in Flume and store them in HDFS for analysis. For a conceptual description of Flume, see the Flume User Guide. This example is a quick walkthrough to get Flume up and running.

Flume Out of the Box

To use Flume in a fresh Quickstart VM:

  1. Import a new VM instance.
  2. Configure the new VM.
    1. Allocate a minimum of 10023 MB memory.
    2. Allocate 2 CPUs.
    3. Allocate 20 MB video memory.
    4. Consider setting the clipboard to bidirectional.
  3. Start the VM.
  4. Launch Cloudera Manager.
  5. In the browser, click the Cloudera Manager link.
  6. Start Hue.
  7. Start Flume.
  8. Use Telnet to test the default Flume implementation.
    1. Open a terminal window.
    2. Install Telnet with the command sudo yum install telnet.
    3. Launch Telnet with the command telnet localhost 10001.
    4. At the prompt, enter Hello world!.
    5. Open /var/log/flume-ng/flume-cmf-flume-AGENT-quickstart.cloudera.log.
    6. Scroll to the bottom of the log, which should have an entry similar to the following. 
      2015-06-05 15:45:55,561 INFO org.apache.flume.sink.LoggerSink: 
Event: { headers:{} body: 48 65 6C 6C 6F 20 77 6F 72 6C 64 21 0D
         Hello world!. }

Writing from Flume to HDFS

You can configure Flume to write incoming messages to data files stored in HDFS for later processing.

To configure Flume to write to HDFS:

  1. In the VM web browser, open Hue.
  2. Click File Browser.
  3. Create the /flume/events directory.
    1. In the /user/cloudera directory, click New->Directory.
    2. Create a directory named flume.
    3. In the flume directory, create a directory named events.
    4. Check the box to the left of the events directory, then click the Permissions setting.
    5. Enable Write access for Group and Other users.
    6. Click Submit.
  4. Change the Flume configuration.
    1. Open Cloudera Manager in your web browser.
    2. In the list of services, click Flume.
    3. Click the Configuration tab.
    4. Scroll or search for the Configuration File item.
    5. Append the following lines to the Configuration File settings. 
      tier1.sinks.sink1.type= HDFS
tier1.sinks.sink1.fileType=DataStream
tier1.sinks.sink1.channel      = channel1
tier1.sinks.sink1.hdfs.path = hdfs://localhost:8020/user/cloudera/flume/events
    6. At the top of the settings list, click Save Changes.
  5. On the far right, choose Actions->Restart to restart Flume.
  6. When the restrart is complete, click Close.
  7. Click the Home tab. If necessary, start the Yarn service.
  8. In a terminal window, launch Telnet with the command telnet localhost 10001.
  9. At the prompt, enter Hello HDFS!.
  10. In the Hue File Browser, open the /user/cloudera/flume/events directory.
  11. There will be a file named FlumeData with a serial number as the file extension. Click the file name link to view the data sent by Flume to HDFS. The output is similar to the following. 
    0000000: 53 45 51 06 21 6f 72 67 2e 61 70 61 63 68 65 2e SEQ.!org.apache.
0000010: 68 61 64 6f 6f 70 2e 69 6f 2e 4c 6f 6e 67 57 72 hadoop.io.LongWr
0000020: 69 74 61 62 6c 65 22 6f 72 67 2e 61 70 61 63 68 itable"org.apach
0000030: 65 2e 68 61 64 6f 6f 70 2e 69 6f 2e 42 79 74 65 e.hadoop.io.Byte
0000040: 73 57 72 69 74 61 62 6c 65 00 00 00 00 00 00 85 sWritable.......
0000050: a6 6f 46 0c f4 16 33 a6 eb 43 c2 21 5c 1b 4f 00 .oF...3..C.!\.O.
0000060: 00 00 18 00 00 00 08 00 00 01 4d c6 1b 01 1f 00 ..........M.....
0000070: 00 00 0c 48 65 6c 6c 6f 20 48 44 46 53 21 0d    ...Hello HDFS!.

Sentiment Analysis of Input from Flume

Now that Flume is sending data to HDFS, you can apply the Sentiment Analysis example to comments you enter.

The source for this example is provided in flumeToHDFS.tar.gz, which contains:
  • flume.config
  • makefile
  • Map.java
  • MrManager.java
  • Reduce.java
  • neg-words.txt
  • pos-words.txt
  • stop-words.txt
  • /shakespeare
    • comedies
    • histories
    • poems
    • tragedies
To test sentiment analysis with Flume input:
  1. Expand flumeToHDFS.tar.gz on the VM.
  2. In a terminal window, go to /flume2hdfs.
  3. Launch Telnet with the command telnet localhost 10001.
  4. Enter the following lines, hitting Enter after each line. (Telnet returns the response OK to each line). 
    I enjoy using CDH. I think CDH is wonderful.
I like the power and flexibility of CDH.
I dislike brussels sprouts. I hate mustard greens.
Flume is a great product. I have several use cases in mind for which it is well suited.
  5. Enter run_flume to start the Sentiment Analysis example via the makefile. The application returns results from all counters, ending with the custom counters and report. 
    
org.myorg.Map$Gauge


NEGATIVE=2


POSITIVE=6
**********
Sentiment score = (6.0 - 2.0) / (6.0 + 2.0)
Sentiment score = 0.5

Positivity score = 6.0/(6.0+2.0)
Positivity score = 75%
**********