Deploying Kafka Connect clusters
You can deploy a Kafka Connect cluster by creating a KafkaConnect resource. The Kafka Connect workers are automatically configured to run in distributed mode. You can configure the number of workers. Each worker is a separate pod.
- Ensure that the Strimzi Cluster Operator is installed and running. See Installation.
- Ensure that you have a working Kafka cluster. The Kafka cluster does not need to be managed by Strimzi, and it does not need to run on Kubernetes.
- The following steps walk you through a basic cluster deployment example. If you want to deploy a Kafka Connect cluster that has third-party connectors or other types of plugins installed, see Installing Kafka Connect connector plugins.
- Ensure that the Secret containing
credentials for the Docker registry where Cloudera Streams Messaging - Kubernetes Operator artifacts are
hosted is available in the namespace where you plan on deploying your cluster. If the secret is
not available, create
it.
kubectl create secret docker-registry [***SECRET NAME***] \ --docker-server [***REGISTRY***] \ --docker-username [***USERNAME***] \ --docker-password [***PASSWORD***] \ --namespace [***NAMESPACE***]
- [***SECRET NAME***] must be the same as the name of the Secret containing registry credentials that you created during Strimzi installation.
- Replace [***REGISTRY***] with the server location of the Docker registry
where Cloudera Streams Messaging - Kubernetes Operator artifacts are hosted. If your Kubernetes cluster
has internet access, use
container.repository.cloudera.com
. Otherwise, enter the server location of your self-hosted registry. - Replace [***USERNAME***] and [***PASSWORD***] with
credentials that provide access to the registry. If you are using
container.repository.cloudera.com
, use your Cloudera credentials. Otherwise, enter credentials providing access to your self-hosted registry.
NAME DESIRED REPLICAS READY
my-connect-cluster 3 True
- Learn more about configuring your Kafka Connect cluster. See Configuring Kafka Connect clusters.
- Install third-party connectors. See Installing Kafka Connect connector plugins.
- Deploy connectors. See Deploying connectors.