HDFS Sink Connector

The HDFS Sink Connector can be used to transfer data from Kafka topics to files on HDFS clusters. Each partition of every topic results in a collection of files named in the following pattern:

{topic name}_{partition number}_{end_offset}.{file extension}

For example, running the HDFS Sink Connector on partition 0 of a topic named sourceTopic can yield the following series of files:

sourceTopic_0_50.avro - for record 0 ~ 50
sourceTopic_0_79.avro - holding record 51 ~ 79
...

The HDFS Sink Connector periodically commits records to final result files. Each commit results in a separate "chunk" file.