ConvertAvroToParquet

Description:

Converts Avro records into Parquet file format. The incoming FlowFile should be a valid avro file. If an incoming FlowFile does not contain any records, an empty parquet file is the output. NOTE: Many Avro datatypes (collections, primitives, and unions of primitives, e.g.) can be converted to parquet, but unions of collections and other complex datatypes may not be able to be converted to Parquet.

Tags:

avro, parquet, convert

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

Display Name	API Name	Default Value	Allowable Values	Description
Compression Type	compression-type	UNCOMPRESSED	UNCOMPRESSED SNAPPY GZIP LZO BROTLI LZ4 ZSTD LZ4_RAW	The type of compression for the file being written.
Row Group Size	row-group-size			The row group size used by the Parquet writer. The value is specified in the format of <Data Size> <Data Unit> where Data Unit is one of B, KB, MB, GB, TB. Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Page Size	page-size			The page size used by the Parquet writer. The value is specified in the format of <Data Size> <Data Unit> where Data Unit is one of B, KB, MB, GB, TB. Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Dictionary Page Size	dictionary-page-size			The dictionary page size used by the Parquet writer. The value is specified in the format of <Data Size> <Data Unit> where Data Unit is one of B, KB, MB, GB, TB. Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Max Padding Size	max-padding-size			The maximum amount of padding that will be used to align row groups with blocks in the underlying filesystem. If the underlying filesystem is not a block filesystem like HDFS, this has no effect. The value is specified in the format of <Data Size> <Data Unit> where Data Unit is one of B, KB, MB, GB, TB. Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Enable Dictionary Encoding	enable-dictionary-encoding		true false	Specifies whether dictionary encoding should be enabled for the Parquet writer
Enable Validation	enable-validation		true false	Specifies whether validation should be enabled for the Parquet writer
Writer Version	writer-version		PARQUET_1_0 PARQUET_2_0	Specifies the version used by Parquet writer

Relationships:

Name	Description
success	Parquet file that was converted successfully from Avro
failure	Avro content that could not be processed

Reads Attributes:

None specified.

Writes Attributes:

Name	Description
filename	Sets the filename to the existing filename with the extension replaced by / added to by .parquet
record.count	Sets the number of records in the parquet file.

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component requires an incoming relationship.

System Resource Considerations:

None specified.