Cloudera Docs

Deployment architecture

Learn the architecture of typical Cloudera Streams Messaging - Kubernetes Operator deployments.

Cloudera Streams Messaging - Kubernetes Operator includes two components that you can install. Strimzi and Cloudera Surveyor. Strimzi enables the deployment and management of Kafka and Kafka Connect clusters on Kubernetes, while Cloudera Surveyor is a UI application designed for monitoring and managing Kafka clusters.

The two components are independent. You can choose to install only Strimzi or only Cloudera Streams Messaging - Kubernetes Operator. Whether you choose to install one or both components depends on your use case and operational objectives. Cloudera recommends installing both.

Strimzi

The following is the typical architecture of a Strimzi and Kafka deployment in Cloudera Streams Messaging - Kubernetes Operator.

Strimzi is installed with a Helm chart. The Helm chart installs various Custom Resource Definitions (CRDs) to the Kubernetes environment. Resources described by these CRDs are managed by the Kubernetes operator applications. Strimzi defines CRDs such as:
  • Kafka – represents a Kafka cluster consisting of Kafka brokers, ZooKeeper servers, Cruise Control and Strimzi Entity Operators.
  • KafkaNodePool – represents a group of Kafka brokers from the cluster that have the same configuration.
  • KafkaTopic – represents a Kafka topic.
  • KafkaUser – represents a user that is external to the Kafka cluster.
  • KafkaRebalance – represents a broker rebalance action for the Kafka cluster.
  • KafkaConnect – represents a Kafka Connect cluster consisting of one or more Kafka Connect workers.
  • KafkaConnector – represents a Kafka Connect connector instance.

When installing the Helm chart, a Strimzi Cluster Operator Kubernetes Deployment is created with a Strimzi Cluster Operator Pod running within the deployment. This application is responsible for monitoring cluster components and reconciling these components when their configuration changes. During installation you are required to register a license, which activates the Strimzi Cluster Operator.

Following installation, you deploy various custom resources in the cluster like Kafka and KafkaNodePool resources. Based on the configuration in the resource, the Strimzi Cluster Operator deploys clusters of the components described by the resources.

Specifically, Kafka and KafkaNodePool resources will deploy Kafka and ZooKeeper clusters. Optionally, if configured, the Kafka resource also deploys the Strimzi Entity Operator and Cruise Control. The Strimzi Entity Operator is responsible for managing other resources inside the particular Kafka cluster (topics, users, and so on), Cruise Control is used for rebalancing Kafka.

KafkaConnect and KafkaConnector resources are used to deploy Kafka Connect clusters and instances of Kafka Connect connectors.

Cloudera Surveyor

The following is the typical architecture of a Cloudera Surveyor deployment in Cloudera Streams Messaging - Kubernetes Operator

Cloudera Surveyor is installed with a Helm chart. The Helm chart creates a Cloudera Surveyor Deployment and Service. Kubernetes creates the associated Cloudera Surveyor Pods as a result of the Deployment. The replica count for the Deployment is set to two by default, resulting in the creation of two Pods for high availability. Optionally, an Ingress is deployed as well if configured. Users access the Cloudera Surveyor UI through the Ingress and Service.