Cloudera Docs

Kafka Connect plugins

Learn what Kafka Connect plugins are shipped with and supported in Cloudera Streams Messaging - Kubernetes Operator.

Connectors

Cloudera Streams Messaging - Kubernetes Operator ships and supports all Kafka Connect connectors included in Apache Kafka.

The full list is as follows.

  • org.apache.kafka.connect.mirror.MirrorCheckpointConnector
  • org.apache.kafka.connect.mirror.MirrorSourceConnector
  • org.apache.kafka.connect.mirror.MirrorHeartBeatConnector
  • org.apache.kafka.connect.file.FileStreamSourceConnector
  • org.apache.kafka.connect.file.FileStreamSinkConnector

Single Message Transforms plugins (transformations and predicates)

Single Message Transforms (SMT) plugins (transformations and predicates) are deployed on top of Kafka Connect connectors. They enable you to apply message transformations and filtering on a single message basis. Cloudera Streams Messaging - Kubernetes Operator ships and supports all transformation and predicates plugins included in Apache Kafka as well as the ConvertFromBytes and ConvertToBytes plugins, which are Cloudera specific plugins.

The full list is as follows.

Transformations
  • com.cloudera.dim.kafka.connect.transforms.ConvertFromBytes
  • com.cloudera.dim.kafka.connect.transforms.ConvertToBytes
  • org.apache.kafka.connect.transforms.Cast
  • org.apache.kafka.connect.transforms.DropHeaders
  • org.apache.kafka.connect.transforms.ExtractField
  • org.apache.kafka.connect.transforms.Filter
  • org.apache.kafka.connect.transforms.Flatten
  • org.apache.kafka.connect.transforms.HeaderFrom
  • org.apache.kafka.connect.transforms.HoistField
  • org.apache.kafka.connect.transforms.InsertField
  • org.apache.kafka.connect.transforms.InsertHeader
  • org.apache.kafka.connect.transforms.MaskField
  • org.apache.kafka.connect.transforms.RegexRouter
  • org.apache.kafka.connect.transforms.ReplaceField
  • org.apache.kafka.connect.transforms.SetSchemaMetadata
  • org.apache.kafka.connect.transforms.TimestampConverter
  • org.apache.kafka.connect.transforms.TimestampRouter
  • org.apache.kafka.connect.transforms.ValueToKey
Predicates
  • org.apache.kafka.connect.transforms.predicates.HasHeaderKey
  • org.apache.kafka.connect.transforms.predicates.RecordIsTombstone
  • org.apache.kafka.connect.transforms.predicates.TopicNameMatches

Converters

Converters can be used to transform Kafka record keys and values between bytes and a specific format. In addition to the JsonConverter, there are converters for most often used primitive types as well.

The full list is as follows.

  • org.apache.kafka.connect.json.JsonConverter
  • org.apache.kafka.connect.converters.ByteArrayConverter
  • org.apache.kafka.connect.converters.BooleanConverter
  • org.apache.kafka.connect.converters.DoubleConverter
  • org.apache.kafka.connect.converters.FloatConverter
  • org.apache.kafka.connect.converters.IntegerConverter
  • org.apache.kafka.connect.converters.LongConverter
  • org.apache.kafka.connect.converters.ShortConverter
  • org.apache.kafka.connect.storage.StringConverter

Header converters

Header converters can be used to transform Kafka record headers between bytes and a specific format. Cloudera Streams Messaging - Kubernetes Operator and Kafka includes a single dedicated header converter, which is the org.apache.kafka.connect.storage.SimpleHeaderConverter.

The SimpleHeaderConverter is the default header converter and is adequate for the majority of use cases. In case your headers are of a specific format, like JSON, you can use any other converter listed in the Converters section as a header converter as well.

Replication policies

A replication policy defines the basic rules of how topics are replicated from source to target clusters when using Kafka Connect-based replication to replicate Kafka data between Kafka clusters.

The full list is as follows.

  • org.apache.kafka.connect.mirror.DefaultReplicationPolicy
  • org.apache.kafka.connect.mirror.IdentityReplicationPolicy