Cloudera Docs

Deploying Kafka Connect clusters

You can deploy a Kafka Connect cluster by creating a KafkaConnect resource. The Kafka Connect workers are automatically configured to run in distributed mode. You can configure the number of workers. Each worker is a separate pod.

  • Ensure that the Strimzi Cluster Operator is installed and running. See Installation.
  • Ensure that you have a working Kafka cluster. The Kafka cluster does not need to be managed by Strimzi, and it does not need to run on Kubernetes.

  • Ensure that a namespace is available where you can deploy your cluster. If not, create one. 

    kubectl create namespace[***NAMESPACE***]
  • Ensure that the Secret containing credentials for the Docker registry where Cloudera Streams Messaging - Kubernetes Operator artifacts are hosted is available in the namespace where you plan on deploying your cluster. If the secret is not available, create it.
    kubectl create secret docker-registry [***SECRET NAME***] \
  --docker-server [***REGISTRY***] \
  --docker-username [***USERNAME***] \
  --docker-password [***PASSWORD***] \
  --namespace [***NAMESPACE***]

    • [***SECRET NAME***] must be the same as the name of the Secret containing registry credentials that you created during Strimzi installation.

    • Replace [***REGISTRY***] with the server location of the Docker registry where Cloudera Streams Messaging - Kubernetes Operator artifacts are hosted. If your Kubernetes cluster has internet access, use container.repository.cloudera.com. Otherwise, enter the server location of your self-hosted registry.

    • Replace [***USERNAME***] and [***PASSWORD***] with credentials that provide access to the registry. If you are using container.repository.cloudera.com, use your Cloudera credentials. Otherwise, enter credentials providing access to your self-hosted registry.

  • The following steps walk you through a basic cluster deployment example. If you want to deploy a Kafka Connect cluster that has third-party connectors or other types of plugins installed, see Installing Kafka Connect connector plugins.
  1. Create a YAML configuration that contains your KafkaConnect resource. 
    apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect-cluster
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  version: 3.9.0.1.3
  replicas: 3
  bootstrapServers: my-cluster-kafka-bootstrap.kafka:9092
    • The spec.version property specifies the Kafka version to use. The property must specify a Cloudera Kafka version supported by Cloudera Streams Messaging - Kubernetes Operator. For example, 3.9.0.1.3. Do not add Apache Kafka versions, they are not supported. You can find a list of supported Kafka versions in the Release Notes.
    • The bootstrapServers property specifies the Kafka brokers to which to connect. Cloudera recommends providing multiple brokers to handle broker failures and enable connecting to another instance.

      The my-cluster-kafka-bootstrap.kafka:9092 value is the bootstrap of a Kafka cluster that is deployed on Kubernetes with Cloudera Streams Messaging - Kubernetes Operator. my-cluster is the name of the cluster specified in metadata.name of the Kafka resource. The kafka-bootstrap string is fixed. The string kafka after the dot is the namespace where the cluster is deployed.

  2. Deploy the resource. 
    kubectl apply --filename [***YAML CONFIG***] --namespace [***NAMESPACE***]

    The namespace where you deploy Kafka Connect must be watched by the Strimzi Cluster Operator.

  3. Verify that the KafkaConnect resource is ready. 
    kubectl get kafkaconnect [***CONNECT CLUSTER NAME***] --namespace [***NAMESPACE***] --watch
If cluster deployment is successful, you should see an output similar to the following.
NAME                 DESIRED REPLICAS   READY
my-connect-cluster   3                  True