Cloudera Docs

Kafka Connect plugins

Learn what Kafka Connect plugins are shipped with and supported in CSM Operator.

Connectors

CSM Operator ships and supports all Kafka Connect connectors included in Apache Kafka.

The full list is as follows.

  • org.apache.kafka.connect.mirror.MirrorCheckpointConnector
  • org.apache.kafka.connect.mirror.MirrorSourceConnector
  • org.apache.kafka.connect.mirror.MirrorHeartBeatConnector
  • org.apache.kafka.connect.file.FileStreamSourceConnector
  • org.apache.kafka.connect.file.FileStreamSinkConnector

Single Message Transforms plugins (transformations and predicates)

Single Message Transforms (SMT) plugins (transformations and predicates) are deployed on top of Kafka Connect connectors. They enable you to apply message transformations and filtering on a single message basis. CSM Operator ships and supports all transformation and predicates plugins included in Apache Kafka. Additionally, CSM Operator ships and supports the ConvertFromBytes and ConvertToBytes plugins, which are Cloudera specific plugins.

The full list is as follows.

Transformations
  • com.cloudera.dim.kafka.connect.transforms.ConvertFromBytes
  • com.cloudera.dim.kafka.connect.transforms.ConvertToBytes
  • org.apache.kafka.connect.transforms.Cast
  • org.apache.kafka.connect.transforms.DropHeaders
  • org.apache.kafka.connect.transforms.ExtractField
  • org.apache.kafka.connect.transforms.Filter
  • org.apache.kafka.connect.transforms.Flatten
  • org.apache.kafka.connect.transforms.HeaderFrom
  • org.apache.kafka.connect.transforms.HoistField
  • org.apache.kafka.connect.transforms.InsertField
  • org.apache.kafka.connect.transforms.InsertHeader
  • org.apache.kafka.connect.transforms.MaskField
  • org.apache.kafka.connect.transforms.RegexRouter
  • org.apache.kafka.connect.transforms.ReplaceField
  • org.apache.kafka.connect.transforms.SetSchemaMetadata
  • org.apache.kafka.connect.transforms.TimestampConverter
  • org.apache.kafka.connect.transforms.TimestampRouter
  • org.apache.kafka.connect.transforms.ValueToKey
Predicates
  • org.apache.kafka.connect.transforms.predicates.HasHeaderKey
  • org.apache.kafka.connect.transforms.predicates.RecordIsTombstone
  • org.apache.kafka.connect.transforms.predicates.TopicNameMatches

Converters

Converters can be used to transform Kafka record keys and values between bytes and a specific format. In addition to the JsonConverter, there are converters for most often used primitive types as well.

The full list is as follows.

  • org.apache.kafka.connect.json.JsonConverter
  • org.apache.kafka.connect.converters.ByteArrayConverter
  • org.apache.kafka.connect.converters.BooleanConverter
  • org.apache.kafka.connect.converters.DoubleConverter
  • org.apache.kafka.connect.converters.FloatConverter
  • org.apache.kafka.connect.converters.IntegerConverter
  • org.apache.kafka.connect.converters.LongConverter
  • org.apache.kafka.connect.converters.ShortConverter
  • org.apache.kafka.connect.storage.StringConverter

Header converters

Header converters can be used to transform Kafka record headers between bytes and a specific format. CSM Operator and Kafka includes a single dedicated header converter, which is the org.apache.kafka.connect.storage.SimpleHeaderConverter.

The SimpleHeaderConverter is the default header converter and is adequate for the majority of use cases. In case your headers are of a specific format, like JSON, you can use any other converter listed in the Converters section as a header converter as well.

Replication policies

A replication policy defines the basic rules of how topics are replicated from source to target clusters when using Kafka Connect-based replication to replicate Kafka data between Kafka clusters.

The full list is as follows.

  • org.apache.kafka.connect.mirror.DefaultReplicationPolicy
  • org.apache.kafka.connect.mirror.IdentityReplicationPolicy