Configure the processor for merging records

Learn how to configure a data target processor for the S3 ingest data flow.

You can merge together multiple record-oriented flow files coming from the ConsumeKafkaRecord_2_0 processor into a large flow file that contains all records of your Kafka data input. This example shows you how to configure the MergeRecord processor.

  1. Launch the Configure Processor window, by right clicking the MergeRecord processor and selecting Configure.
    This gives you a configuration dialog with the following tabs: Settings, Scheduling, Properties, Comments.
  2. Configure the processor according to the behavior you expect in your data flow.
  3. When you have finished configuring the options you need, save the changes by clicking the Apply button.

    Make sure that you set all required properties, as you cannot start the processor until all mandatory properties have been configured.

    In this example the following settings and properties are used:

    Table 1. MergeRecord processor scheduling
    Scheduling Description Example value for ingest data flow
    Automatically Terminate Relationships original
    Table 2. MergeRecord processor properties
    Property Description Example value for ingest data flow

    RecordReader

    Specify the Controller Service to use for reading incoming data.

    ReadCustomerCSVRoot

    RecordWriter

    Specify the Controller Service to use for writing out the records.

    CSVRecordSetWriter

    Merge Strategy

    Specify the algorithm used to merge records.

    The Bin-Packing Algorithm generates a FlowFile populated by arbitrarily chosen FlowFiles.

    Bin-Packing Algorithm

    Minimum Number of Records

    Specify the minimum number of records to include in a bin.

    900

    Maximum Number of Records

    Specify the maximum number of Records to include in a bin.

    1000

    For a complete list of MergeRecord properties, see the processor documentation.

Configure the processor for your data target.