ConvertToBytes

ConvertToBytes is a Cloudera specific transformation plugin that converts Kafka Connect internal data to binary data. You can use this plugin to make connectors that only support binary data compatible with the Single Message Transforms (SMT) framework.

Fully qualified names

com.cloudera.dim.kafka.connect.transformations.convert.ConvertToBytes$Key
com.cloudera.dim.kafka.connect.transformations.convert.ConvertToBytes$Value

Description

The ConvertToBytes transformation plugin accepts data in the Kafka Connect internal data format and converts it to binary data with a nested converter. To support header based converter logic, for example when using the AvroConverter with the schema ID encoded in the record header, the plugin requires a header converter to correctly transform record headers when interacting with the converter. This plugin supports both key and value conversion.

Using this plugin with connectors that only support binary data makes the connector fully compatible with the SMT framework. On their own, connectors that only support binary data, for example the Stateless NiFi Sink connector, have limited compatibility with transformations even if the binary data is structured. This is because the format of the data after transformations are carried out is normally the Kafka Connect internal data format. Binary only connectors, however, expect data that has the BYTES schema and do not provide conversion from the Kafka Connect internal data format by default. When you use the ConvertToBytes plugin with a binary only connector, the structured data is converted to binary format, which can then be picked up by the connector.

Example

The following configuration example adds a ConvertToBytes transformation as the last step of the transformation chain. The conversion uses a schemaless JSON transformation to serialize the structured data. The transformation steps, the connector, or the converter, whichever comes directly after ToBytes, receives a properly structured record instead of binary data.

{  
    	"transforms" : "...,ToBytes",
    	"transforms.ToBytes.type" : "com.cloudera.dim.kafka.connect.transformations.convert.ConvertToBytes$Value",
    	"transforms.ToBytes.converter" : "org.apache.kafka.connect.json.JsonConverter",
    	"transforms.ToBytes.converter.schemas.enable" : "false"
}

Configuration properties

Table 1. ConvertToBytes properties reference
Property	Default Value	Required	Description
converter		True	The fully qualified name of the converter implementation to use. For example: `com.cloudera.dim.kafka.connect.converts.AvroConverter`
header.converter	`org.apache.kafka.connect.storage.SimpleHeaderConverter`	True	The fully qualified name of the header converter implementation to use. This converter must match the header converter of the connector.
converter.		False	A configuration prefix. Use this prefix to configure the properties of the converter specified in `converter`. Property keys and values specified with the prefix are passed directly to the converter with the prefix stripped. For example: `transforms.[*TRANSFORM ALIAS].converter.[CONVERTER PROPERTY KEY]:[CONVERTER PROPERTY VALUE*]`
header.converter.		False	A configuration prefix. Use this prefix to configure the properties of the header converter specified in `header.converter`. Property keys and values specified with the prefix are passed directly to the header converter with the prefix stripped. For example: `transforms.[*TRANSFORM ALIAS].converter.[HEADER CONVERTER PROPERTY KEY]:[HEADER CONVERTER PROPERTY VALUE*]`