Amazon S3 Sink Connector Properties Reference

Amazon S3 Sink Connector Properties Reference.

The following table collects connector properties that are specific for the Amazon S3 Sink Connector. For properties common to all sink connectors, see the upstream Apache Kafka documentation.

Property Name Description Type Default Value Accepted Values Recommended Value
aws.s3.bucket The target S3 bucket name. String none Any valid S3 bucket name.
aws.s3.service_endpoint The target S3 host and port. String none Any valid S3 endpoint.
aws.access_key_id The AWS secret key ID to authenticate. String none Any valid secret key issued by AWS.
aws.secret_access_key The AWS secret access key to authenticate. String none Any valid access key issued by AWS.
value.converter Value conversion class. String none com.cloudera.dim.kafka.connect.converts.AvroConverter
value.converter.passthrough.enabled Configures whether the AvroConverter translates an Avro record into Kafka Connect Data or transparently passes the Avro encoded bytes as payload. Boolean true true, false True if input and output are both Avro.
value.converter.schema.registry.url The URL to the Schema Registry server. String none The S3 storage implementation class. String none com.cloudera.dim.kafka.connect.s3.S3PartitionStorage
output.writer The output file writer which determines the type of file to be written. The value of this property should be the FQCN of a class that implements the PartitionWriter interface. String none
  • com.cloudera.dim.kafka.connect.partition.writers.avro.AvroPartitionWriter
  • com.cloudera.dim.kafka.connect.partition.writers.json.JsonPartitionWriter
  • com.cloudera.dim.kafka.connect.hdfs.parquet.ParquetPartitionWriter
  • com.cloudera.dim.kafka.connect.partition.writers.txt.TxtPartitionWriter

Configures Whether the output writer expects an Avro encoded Kafka Connect data record. Must match the configuration ofvalue.converter.passthrough.enabled.

Boolean none true, false True if input and output are both Avro.