Stream Data Using NiFi

NiFi provides a highly intuitive streaming user interface that is compatible with most types of data sources.

1. Drag the icon to your workspace.
   NiFi displays the Add Processor dialog box.
2. Select the TailFile type of processor and click Add.
   NiFi displays a new TailFile processor:
   
3. Right-click (processor icon) and select Configure to display the Configure Processor dialog box:
   1. In the Settings tab, change the name to Ingest $DATASOURCE Events:
      
   2. In the Properties tab, enter the path to the data source file in the Value column for the File(s) to Tail property:

      
      
      

4. Add another processor by dragging the Processor icon to your workspace.
5. Select the PutKafka type of processor and click Add.
6. Right-click the processor and select Configure.
7. In the Settings tab, change the name to Stream to Metron and then select the relationship check boxes for failure and success.

   
   
   

8. In the Properties tab, set the following three properties:

   Known Brokers

   $KAFKA_HOST:6667

   Topic Name

   $DATAPROCESSOR

   Client Name

   nifi-$DATAPROCESSOR

   
   
   

9. Create a connection by dragging the arrow from the Ingest $DATAPROCESSOR Events processor to the Stream to Metron processor.
   NiFi displays Create Connection dialog box.

   
   
   

10. Click Add to accept the default settings for the connection.
11. Press Shift and draw a box around both parsers to select the entire flow; then click the green arrow.
    All of the processor icons turn into green arrows:

    
    
    

12. In the Operate panel, click the arrow icon.

    
    
    

13. Generate some data using the new data processor client.
14. Look at the Storm UI for the parser topology and confirm that tuples are coming in.
15. After about five minutes, you see a new index called $DATAPROCESSOR_index* in either the Solr Admin UI or the Elastic Admin UI.