ConvertToBytes
ConvertToBytes is a Cloudera specific transformation plugin that converts Kafka Connect internal data to binary data. You can use this plugin to make connectors that only support binary data compatible with the Single Message Transforms (SMT) framework.
Fully qualified names
com.cloudera.dim.kafka.connect.transformations.convert.ConvertToBytes$Key
com.cloudera.dim.kafka.connect.transformations.convert.ConvertToBytes$Value
Description
The ConvertToBytes
transformation plugin accepts data in the Kafka Connect
internal data format and converts it to binary data with a nested converter. To support header
based converter logic, for example when using the AvroConverter
with the schema
ID encoded in the record header, the plugin requires a header converter to correctly transform
record headers when interacting with the converter. This plugin supports both key and value
conversion.
Using this plugin with connectors that only support binary data makes the connector fully
compatible with the SMT framework. On their own, connectors that only support binary data, for
example the Stateless NiFi Sink connector, have limited compatibility with transformations even
if the binary data is structured. This is because the format of the data after transformations
are carried out is normally the Kafka Connect internal data format. Binary only connectors,
however, expect data that has the BYTES schema and do not provide conversion from the Kafka
Connect internal data format by default. When you use the ConvertToBytes
plugin
with a binary only connector, the structured data is converted to binary format, which can then
be picked up by the connector.
Example
The following configuration example adds a ConvertToBytes transformation as
the last step of the transformation chain. The conversion uses a schemaless JSON transformation
to serialize the structured data. The transformation steps, the connector, or the converter,
whichever comes directly after ToBytes
, receives a properly structured record
instead of binary data.
{
"transforms" : "...,ToBytes",
"transforms.ToBytes.type" : "com.cloudera.dim.kafka.connect.transformations.convert.ConvertToBytes$Value",
"transforms.ToBytes.converter" : "org.apache.kafka.connect.json.JsonConverter",
"transforms.ToBytes.converter.schemas.enable" : "false"
}
Configuration properties
Property | Default Value | Required | Description |
---|---|---|---|
converter | True | The fully qualified name of the converter implementation to use. For example:
com.cloudera.dim.kafka.connect.converts.AvroConverter |
|
header.converter | org.apache.kafka.connect.storage.SimpleHeaderConverter |
True | The fully qualified name of the header converter implementation to use. This converter must match the header converter of the connector. |
converter. | False | A configuration prefix. Use this prefix to configure the properties of the converter
specified in converter . Property keys and values specified with the prefix
are passed directly to the converter with the prefix stripped. For example:
|
|
header.converter. | False | A configuration prefix. Use this prefix to configure the properties of the header
converter specified in header.converter . Property keys and values specified
with the prefix are passed directly to the header converter with the prefix stripped. For
example:
|