The processor generates N flow files from the input, and adds attributes with the offsets required to read the group of rows in the FlowFile's content. Can be used to increase the overall efficiency of processing extremely large Parquet files.
parquet, split, partition, break apart, efficient processing, load balance, cluster
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name | API Name | Default Value | Allowable Values | Description |
|---|---|---|---|---|
| Records Per Split | Records Per Split | Specifies how many records should be covered in each FlowFile Supports Expression Language: true (will be evaluated using flow file attributes and variable registry) | ||
| Zero Content Output | Zero Content Output | false |
| Whether to do, or do not copy the content of input FlowFile. |
| Name | Description |
|---|---|
| success | FlowFiles, with special attributes that represent a chunk of the input file. |
| Name | Description |
|---|---|
| record.offset | Gets the index of first record in the input. |
| record.count | Gets the number of records in the input. |
| parquet.file.range.startOffset | Gets the start offset of the selected row group in the parquet file. |
| parquet.file.range.endOffset | Gets the end offset of the selected row group in the parquet file. |
| Name | Description |
|---|---|
| record.offset | Sets the index of first record of the parquet file. |
| record.count | Sets the number of records in the parquet file. |