Installing Kafka in Cloudera on cloud

Learn how Kafka is installed and set up in Cloudera on cloud with the Streams Messaging cluster templates.

In Cloudera on cloud, Kafka is automatically deployed on clusters that you provision with the Streams Messaging Light Duty, Heavy Duty, and High Availability templates available in the Cloudera Data Hub service.

Kafka Broker roles are deployed across two host groups: Core Broker and Broker. The Core Broker host group contains the minimum required brokers for the cluster, while the Broker host group is used for broker scaling.

Kafka Connect roles are deployed by default in the Light Duty template and are colocated with Broker roles. In the Heavy Duty and High Availability templates, Connect roles have their dedicated host group, but the dedicated host group is not deployed by default. This means that in Cloudera on cloud explicit configuration is required at or after cluster deployment for the cluster to include Kafka Connect functionality.

KRaft Controller roles are deployed in all templates in a separate host group called KRaft that has its default instance count set to 0. This means that all Kafka instances deployed in Cloudera on cloud with default configuration run in ZooKeeper mode. To use KRaft, you must explicitly set the KRaft host group instance count to at least one during cluster deployment or migrate existing ZooKeeper-based clusters to KRaft after provisioning.

MirrorMaker roles are not deployed with the Streams Messaging templates. Instead the templates deploy Streams Replication Manager, which you can use to replicate data across different Kafka clusters.

For additional information on the templates, a step-by-step deployment guide using the templates, as well as scaling, see the Related Information section.