Cloudera Docs

Build the data flow

Learn how you can create an ingest data flow to move data to an Iceberg table in Hive or Impala. This involves using NiFi in your Flow Management cluster. The example data flow generates FlowFiles with random CSV data and writes this data into an Iceberg table.

  1. Open NiFi 
in 
Data Hub.
    1. To access the NiFi service in your Flow Management 
Data Hub cluster, navigate to Management Console service > Data Hub Clusters.
    2. Click the tile representing the Flow Management Data Hub cluster you want to work with.
    3. Click the NiFi icon in the Services section of the cluster overview page to access the NiFi UI.
      You will be logged into NiFi automatically with your CDP credentials.
  2. Add the GenerateFlowFile processor for data input.
    This processor creates FlowFiles with random data or custom content.
    1. Drag and drop the processor icon into the canvas. This displays a dialog that allows you to choose the processor you want to add.
    2. Select the GenerateFlowFile processor from the list.
    3. Click Add or double-click the required processor type to add it to the canvas.
    You will configure the GenerateFlowFile processor to define how to create the sample data in Configure the processor for your data source.
  3. Add the PutIceberg processor for data output.
    You will configure the PutIceberg processor in Configure the processor for your data target.
  4. Connect the two processors to create a flow.
    1. Drag the connection icon from the first processor, and drop it on the second processor.

      A Create Connection dialog appears with two tabs: Details and Settings.

    2. Configure the connection.

      You can configure the connection's name, flowfile expiration time period, thresholds for back pressure, load balance strategy and prioritization.

    3. Click Add to close the dialog box and add the connection to your data flow.
  5. Optionally, you can add success and failure funnels to your data flow, which help you see where flow files are being routed when your flow is running. Connect the PutIceberg to these funnels with success and failure connections.
    If you want to know more about working with funnels, see the Apache NiFi User Guide.
Your data flow will look similar to the following:
Create controller services for your data flow. You will need these services when configuring the PutIceberg processor.