HDFS Spout Example

The following example creates an HDFS spout that reads text files from HDFS path hdfs://localhost:54310/source.

// Instantiate spout to read text files
HdfsSpout textReaderSpout = newHdfsSpout().setReaderType("text")
                                          .withOutputFields(TextFileReader.defaultFields)                                      
                                          .setHdfsUri("hdfs://localhost:54310")  // reqd
                                          .setSourceDir("/data/in")              // reqd                                      
                                          .setArchiveDir("/data/done")           // reqd
                                          .setBadFilesDir("/data/badfiles");     // required                                      

// If using Kerberos
HashMap hdfsSettings = new HashMap();
hdfsSettings.put("hdfs.keytab.file", "/path/to/keytab");
hdfsSettings.put("hdfs.kerberos.principal","user@EXAMPLE.com");

textReaderSpout.setHdfsClientSettings(hdfsSettings);

// Create topology
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("hdfsspout", textReaderSpout, SPOUT_NUM);

// Set up bolts and wire up topology
   ...

// Submit topology with config
Config conf = new Config();
StormSubmitter.submitTopologyWithProgressBar("topologyName", conf, builder.createTopology());

A sample topology HdfsSpoutTopology is provided in the storm-starter module.