Configuring connectors

Learn how you configure connectors with KafkaConnector resources. Configuring connectors with Kafkaconnector resources is the recommended method by Cloudera for configuring connectors.

Connectors that you deploy using KafkaConnector resources are configured with their corresponding KafkaConnector resource. When you make a configuration update, the Strimzi Cluster Operator, which manages the lifecycle of connectors, updates configurations.

Connector properties that you configure in your KafkaConnector resources largely depend on the specific connector you are using. This is because most configuration properties are connector specific. Always consult the documentation of the specific connector that you want to configure.

Configuring automatic restarts for connectors

You can enable the automatic restart of failed connectors and tasks with spec.autoRestart.enabled. Additionally, you can configure the maximum number of allowed automatic restarts with spec.autoRestart.maxRestarts. Both properties are configured in the KafkaConnector resource.

#...
kind: KafkaConnector
spec:
  autoRestart:
    enabled: true
    maxRestarts: 10

Configuring connector properties

You configure a connector by specifying connector properties in spec.config of the KafkaConnector resource.

This is an example configuration for a FilestreamSourceConnector that reads the Apache Kafka license file and produces the contents of the file to a specified topic.

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: my-source-connector
  labels:
    strimzi.io/cluster: my-connect-cluster
spec:
  class: org.apache.kafka.connect.file.FileStreamSourceConnector
  tasksMax: 2
  config:
    file: "/opt/kafka/LICENSE"
    topic: my-topic

Connector configurations you specify in spec.config will largely depend on the connector itself. Consult the documentation for your specific connector to learn what properties it supports.

Configuring client overrides in connectors

Learn how to configure client configuration overrides in connectors. Configuring overrides enables you to fine-tune your connector configuration.

The Kafka Connect framework manages Kafka clients (producers, consumers, and admin clients) used by connectors and tasks. By default, these clients use worker-level properties. In some use-cases, you might want to fine-tune these properties with overrides.

For example, you can use overrides to configure unique authentication credentials for your connectors. Or you can use overrides to fine-tune connector performance.

Overrides take precedence over worker-level properties. You configure overrides with the following prefixes.

producer.override. – used for overriding producer properties.
consumer.override. – used for overriding consumer properties.
admin.override. – used for overriding admin client properties.

To configure an override, add any supported consumer, producer, or admin client property with the corresponding prefix to spec.config in your KafkaConnector resource.

#...
kind: KafkaConnector
spec:
  config:
    producer.override.batch.size: 1234

This example configures the producer batch size, which is a typical property you tweak when fine-tuning connectors and clients for performance.