Cloudera Docs

Build the data flow

Learn how you can create an ingest data flow to move data from Apache Kafka to Apache Solr. This involves adding processors and other data flow elements to the NiFi canvas, configuring them, and connecting the elements to create the data flow.

Use the PutSolrRecord processor to build your Solr ingest data flow.

  1. Open NiFi 
in 
CDP Public Cloud.
    1. To access the NiFi service in your Flow Management cluster, navigate to Management Console > Data Hub Clusters.
    2. Click the tile representing the Flow Management cluster you want to work with.
    3. Click NiFi in the Services section of the cluster overview page to access the NiFi UI.
    You will be logged into NiFi automatically with your CDP credentials.
  2. Add the NiFi processors to your canvas.
    1. Select the processor icon from the Cloudera Flow Management actions pane, and drag it to the canvas.
    2. Use the Add Processor filter box to search for the processors you want to add.
    3. Select the following processor and click Add.
      • ConsumeKafkaRecord_2_0
      • PutSolrRecord
  3. Connect the two processors to create the data flow.
    1. Click the connection icon in the first processor, and drag it to the second processor.

      You can see the Create Connection dialog with two tabs: Details and Settings. You can configure the connection's name, flowfile expiration time period, thresholds for back pressure, load balance strategy, and prioritization.

    2. Click Add to close the dialog box and add the connection to your flow.
    3. Optional: You can add success and failure funnels to your data flow, which helps you to see where flow files are routed when your data flow is running.

      For more information about working with funnels, see the Apache NiFi Documentation.

Your data flow may look similar to the following:
Create a controller service for your data flow. You will need this service later as you configure your data flow target processor.