Cloudera Docs

Deploying Kafka Connect clusters

You can deploy a Kafka Connect cluster by creating a KafkaConnect resource. The Kafka Connect workers are automatically configured to run in distributed mode. You can configure the number of workers. Each worker is a separate pod.

  • Ensure that the Strimzi Cluster Operator is installed and running. See Installation.
  • Ensure that you have a working Kafka cluster. The Kafka cluster does not need to be managed by Strimzi, and it does not need to run on Kubernetes.
  • The following steps walk you through a basic cluster deployment example. If you want to deploy a Kafka Connect cluster that has third-party connectors or other types of plugins installed, see Installing Kafka Connect connector plugins.
  • Ensure that the Secret containing your Cloudera credentials is available in the namespace where you plan on deploying your Kafka Connect cluster. If the secret is not available, create it.
    kubectl create secret docker-registry [***SECRET NAME***] \
  --docker-server container.repository.cloudera.com \
  --docker-username [***USERNAME***] \
  --docker-password [***PASSWORD***] \
  --namespace [***NAMESPACE***]
    • Replace [***USERNAME***] and [***PASSWORD***] with your Cloudera credentials.
    • [***SECRET NAME***] must be the same as the name of the Secret that contains Cloudera credentials and exists in the Strimzi Cluster Operator namespace.
  1. Create a YAML configuration that contains your KafkaConnect resource. 
    apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect-cluster
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  version: 3.7.0.1.1
  replicas: 3
  bootstrapServers: my-cluster-kafka-bootstrap.kafka:9092
    • The spec.version property specifies the Kafka version to use. The property must specify a Cloudera Kafka version supported by CSM Operator. For example, 3.7.0.1.1. Do not add Apache Kafka versions, they are not supported. You can find a list of supported Kafka versions in the Release Notes.
    • The bootstrapServers property specifies the Kafka brokers to which to connect. Cloudera recommends providing multiple brokers to handle broker failures and enable connecting to another instance.
  2. Deploy the resource. 
    kubectl apply --filename [***YAML CONFIG***] --namespace [***NAMESPACE***]

    The namespace where you deploy Kafka Connect must be watched by the Strimzi Cluster Operator.

  3. Verify that the KafkaConnect resource is ready. 
    kubectl get kafkaconnect [***CONNECT CLUSTER NAME***] --namespace [***NAMESPACE***] --watch
If cluster deployment is successful, you should see an output similar to the following.
NAME                 DESIRED REPLICAS   READY
my-connect-cluster   3                  True