HDFS Sink Connector Properties Reference

HDFS Sink Connector Properties Reference.

The following table collects connector properties that are specific for the HDFS Sink Connector. For properties common to all sink connectors, see the upstream Apache Kafka documentation.

Property Name Description Type Default Value Accepted Values Recommended Values
hdfs.uri The file system URI to connect to on the destination cluster. This property supports any valid HDFS or o3fs URI. String None
hdfs.output The root directory on the HDFS cluster where all the output files will reside. The sub path has the following pattern: {topic}/{topic}_{partition}_{endoffset}.{file extension} String /tmp Any path on the HDFS file system where the role has read write permission.
hdfs.kerberos.authentication Enables or disables secure access to the HDFS cluster by authenticating with Kerberos. Boolean false true or false
hdfs.kerberos.user.principal The kerberos user principal. String null The host-dependent Kerberos principal assigned to the Kafka Connect role.
hdfs.kerberos.keytab.path The path to the Kerberos keytab file. String null In a Cloudera Manager provisioned environment, it’s recommended to use the Cloudera Manager Config Provider to automatically provision the path.
hdfs.kerberos.namenode.principal The kerberos name node principal. Required when the HDFS cluster has data encryption on. String null
hadoop.conf.path The path to the site specific Hadoop configuration XML files. Required when the HDFS cluster has data encryption on. String null
output.writer The output file writer which determines the type of file to be written to the HDFS cluster. The value of this property should be the FQCN of a class that implements the PartitionWriter interface. String null
  • com.cloudera.dim.kafka.connect.partition.writers.avro.AvroPartitionWriter
  • com.cloudera.dim.kafka.connect.partition.writers.json.JsonPartitionWriter
  • com.cloudera.dim.kafka.connect.hdfs.parquet.ParquetPartitionWriter
  • com.cloudera.dim.kafka.connect.partition.writers.txt.TxtPartitionWriter
com.cloudera.dim.kafka.connect.partition.writers.avro.AvroPartitionWriter
output.avro.passthrough.enabled Configures whether the output writer expects an Avro encoded Kafka Connect data record. Must match the configuration of value.converter.passthrough.enabled. Boolean true true or false True if input and output are both Avro.
value.converter The converter to be used to translate the value field of the source Kafka record into Kafka Connect Data format. String Inherited from Kafka Connect worker properties.
  • org.apache.kafka.connect.json.JsonConverter
  • org.apache.kafka.connect.storage.StringConverter
  • com.cloudera.dim.kafka.connect.converts.AvroConverter
com.cloudera.dim.kafka.connect.converts.AvroConverter
value.converter.schema.registry.url The URL to the Schema Registry server. String null true or false
value.converter.passthrough.enabled Configures whether the AvroConverter translates an Avro record into Kafka Connect Data or transparently passes the Avro encoded bytes as payload. Boolean true true or false True if input and output are both Avro.