Cloudera Docs

Configure your source processor

You can set up a dataflow to push data into Snowflake database tables from many different locations. To do this, start by configuring the Processor four your data source by launching the Configure Processor window and specifying the necessary configurations.

Configure GenerateFlowFile to create random data for this example dataflow. GenerateFlowFile is useful when you are testing or creating proof of concept dataflows. When you have confirmed that this dataflow meets your business use case, you will likely replace it with a processor getting data from your actual data source.

See Related Information for full details on this Apache NiFi Processor.

  • You have built the dataflow.
  • You have configured your Controller Services.
  1. Launch the Configure Processor window, by right clicking the GenerateFlowFile processor and selecting Configure. A configuration dialog with the following tabs is displayed: Settings, Scheduling, Properties, and Comments.
  2. Configure the processor according to the behavior you expect in your dataflow.
    See the Example section below for recommended configuration to satisfy this example use case.
  3. When you have finished configuring the options you need, save the changes by clicking the Apply button.

The following settings and properties are used in this example:

Table 1. GenerateFlowFile processor scheduling
Scheduling Description Example value for ingest data flow

Run Schedule

Run schedule dictates how often the processor should be scheduled to run. The valid values for this field depend on the selected Scheduling Strategy.

60 s
Table 2. GenerateFlowFile processor properties
Description Example value for ingest data flow

Custom text

If Data Format is text and if Unique FlowFiles is false, you can provide custom to be used as the content of the generated FlowFiles.

The expression statement in the example value generates a random ID between 1 and 10 000, with random last names assigned.

 

R_REGIONKEY, R_NAME, R_COMMENT 
100,foo1, blablabla 
101, foo2, blabla 
102, foo3, bla

Once you have configured your Controller Services, proceed to the following tasks:

  • Configure your target Processor.
  • Confirm your dataflow success.