Cloudera Docs

Deploying Kafka Connect clusters

You can deploy a Kafka Connect cluster by creating a KafkaConnect resource. The Kafka Connect workers are automatically configured to run in distributed mode. You can configure the number of workers. Each worker is a separate pod.

  • Ensure that the Strimzi Cluster Operator is installed and running. See Installation.
  • Ensure that you have a working Kafka cluster. The Kafka cluster does not need to be managed by Strimzi, and it does not need to run on Kubernetes.
  • The following steps walk you through a basic cluster deployment example. If you want to deploy a Kafka Connect cluster that has third-party connectors or other types of plugins installed, see Installing Kafka Connect connector plugins.
  • Ensure that the Secret containing credentials for the Docker registry where CSM Operator artifacts are hosted is available in the namespace where you plan on deploying your cluster. If the secret is not available, create it.
    kubectl create secret docker-registry [***SECRET NAME***] \
  --docker-server [***REGISTRY***] \
  --docker-username [***USERNAME***] \
  --docker-password [***PASSWORD***] \
  --namespace [***NAMESPACE***]
    • [***SECRET NAME***] must be the same as the name of the Secret containing registry credentials that you created during Strimzi installation.
    • Replace [***REGISTRY***] with the server location of the Docker registry where CSM Operator artifacts are hosted. If your Kubernetes cluster has internet access, use container.repository.cloudera.com. Otherwise, enter the server location of your self-hosted registry.
    • Replace [***USERNAME***] and [***PASSWORD***] with credentials that provide access to the registry. If you are using container.repository.cloudera.com, use your Cloudera credentials. Otherwise, enter credentials providing access to your self-hosted registry.
  1. Create a YAML configuration that contains your KafkaConnect resource. 
    apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect-cluster
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  version: 3.7.0.1.1
  replicas: 3
  bootstrapServers: my-cluster-kafka-bootstrap.kafka:9092
    • The spec.version property specifies the Kafka version to use. The property must specify a Cloudera Kafka version supported by CSM Operator. For example, 3.7.0.1.1. Do not add Apache Kafka versions, they are not supported. You can find a list of supported Kafka versions in the Release Notes.
    • The bootstrapServers property specifies the Kafka brokers to which to connect. Cloudera recommends providing multiple brokers to handle broker failures and enable connecting to another instance.
  2. Deploy the resource. 
    kubectl apply --filename [***YAML CONFIG***] --namespace [***NAMESPACE***]

    The namespace where you deploy Kafka Connect must be watched by the Strimzi Cluster Operator.

  3. Verify that the KafkaConnect resource is ready. 
    kubectl get kafkaconnect [***CONNECT CLUSTER NAME***] --namespace [***NAMESPACE***] --watch
If cluster deployment is successful, you should see an output similar to the following.
NAME                 DESIRED REPLICAS   READY
my-connect-cluster   3                  True