Setting up Kafka Connect in Cloudera on cloud
Learn how you can set up Kafka Connect in Cloudera on cloud with the Streams Messaging cluster templates.
Kafka Connect in Cloudera is provided in the form of a Kafka
service role. The role is called Kafka Connect. In Cloudera on cloud, Kafka Connect can be deployed with the Streams
Messaging Light Duty, Heavy Duty, and High Availability templates available in the Cloudera Data Hub service. However, there are some differences between how
Kafka Connect is deployed with each of the templates.
- Light Duty
- In the Streams Messaging Light Duty template, Kafka Connect roles share a host group and nodes with other service roles. Specifically, Kafka Connect is deployed in the Broker and Core_broker host groups and shares nodes with Kafka broker, ZooKeeper, and Streams Replication Manager Driver roles. The Core_broker is a mandatory host group in Light Duty clusters and has an instance (node) count of three. This means that by default, all Light Duty cluster you deploy will include a minimum of three Kafka Connect roles. If required, Kafka Connect can also be scaled in Light Duty clusters by scaling the Broker host group.
- Heavy Duty and High Availability
- In the Heavy Duty and High Availability templates, Kafka Connect roles are deployed in a dedicated host group named Connect. The instance count of this host group is set to zero by default. This means that, unless specifically provisioned, Kafka Connect roles are not deployed in Heavy Duty or High Availability clusters by default. The instance (node) count of the Connect host group can be configured in Cloudera Data Hub during cluster provisioning. Additionally, you can add Kafka Connect to already running Heavy Duty or High Availability cluster by scaling the Connect host group.
Streams Messaging Manager integration
Kafka Connect deployed in Cloudera on cloud is by default configured to integrate with the Streams Messaging Manager instance running in the cluster. This means that, if you provisioned your cluster with Kafka Connect, you will be able to manage, monitor, and deploy Kafka Connect connectors with Streams Messaging Manager as soon as the cluster is deployed with no additional configuration required. Kafka Connect can be managed in Streams Messaging Manager on the Connect page. For more information, see Monitoring Kafka Connect using Streams Messaging Manager.
