ConvertFromBytes

ConvertFromBytes is a Cloudera specific transformation plugin that converts binary data to the Kafka Connect internal data format. You can use this plugin to make connectors that only support binary data compatible with the Single Message Transforms (SMT) framework.

Fully qualified names

com.cloudera.dim.kafka.connect.transformations.convert.ConvertFromBytes$Key
com.cloudera.dim.kafka.connect.transformations.convert.ConvertFromBytes$Value

Description

The ConvertFromBytes transformation plugin accepts binary data and converts it into the Kafka Connect internal data format with a nested converter that transforms binary data. To support header based converter logic, for example when using the AvroConverter with the schema ID encoded in the record header, the plugin requires a header converter to correctly transform record headers when interacting with the converter. This plugin supports both key and value conversion.

Using this plugin with connectors that only support binary data makes the connector fully compatible with the SMT framework. On their own, connectors that only support binary data have limited compatibility with transformations even if the binary data is structured. This is because transformations are only fully supported on data that is in the Kafka Connect internal data format. Binary only connectors like the Mirror Source (MirrorSourceConnector) or the Stateless NiFi Source connectors emit data that has the BYTES schema and do not provide conversion to the Kafka Connect internal data format by default. When you use a connector such as these with the ConvertFromBytes plugin, the binary data is parsed into a compatible structure, which can then be further processed with the transformation chain.

Example

The following configuration example adds a ConvertFromBytes transformation as a first step of the transformation chain. The conversion uses a schemaless JSON transformation to parse the binary data. The transformation steps, the connector, or the converter, whichever comes directly after FromBytes, receives a properly structured record instead of binary data.

{  
    	"transforms" : "FromBytes,...",
    	"transforms.FromBytes.type" : "com.cloudera.dim.kafka.connect.transformations.convert.ConvertFromBytes$Value",
    	"transforms.FromBytes.converter" : "org.apache.kafka.connect.json.JsonConverter",
    	"transforms.FromBytes.converter.schemas.enable" : "false"
}

Configuration properties

Table 1. ConvertFromBytes properties reference
Property	Default Value	Required	Description
converter		True	The fully qualified name of the converter implementation to use. For example: `com.cloudera.dim.kafka.connect.converts.AvroConverter`
header.converter	`org.apache.kafka.connect.storage.SimpleHeaderConverter`	True	The fully qualified name of the header converter implementation to use. This converter must match the header converter of the connector.
converter.		False	A configuration prefix. Use this prefix to configure the properties of the converter specified in `converter`. Property keys and values specified with the prefix are passed directly to the converter with the prefix stripped. For example: `transforms.[*TRANSFORM ALIAS].converter.[CONVERTER PROPERTY KEY]:[CONVERTER PROPERTY VALUE*]`
header.converter.		False	A configuration prefix. Use this prefix to configure the properties of the header converter specified in `header.converter`. Property keys and values specified with the prefix are passed directly to the header converter with the prefix stripped. For example: `transforms.[*TRANSFORM ALIAS].converter.[HEADER CONVERTER PROPERTY KEY]:[HEADER CONVERTER PROPERTY VALUE*]`