Kafka Connect Setup [Technical Preview]
Learn how you can set up Kafka Connect in CDP Public Cloud with the Streams Messaging cluster templates.
Kafka Connect in CDP is provided in the form of a Kafka service
role. The role is called Kafka Connect. In CDP Public Cloud, Kafka Connect can be deployed
with the Streams Messaging Light Duty, Heavy Duty, and High Availability templates available
in the Data Hub service. However, there are some differences between how Kafka Connect is
deployed with each of the templates.
- Light Duty
- In the Streams Messaging Light Duty template, Kafka Connect roles share a host group and nodes with other service roles. Specifically, Kafka Connect is deployed in the Broker and Core_broker host groups and shares nodes with Kafka broker, ZooKeeper, and SRM Driver roles. The Core_broker is a mandatory host group in Light Duty clusters and has an instance (node) count of three. This means that by default, all Light Duty cluster you deploy will include a minimum of three Kafka Connect roles. If required, Kafka Connect can also be scaled in Light Duty clusters by scaling the Broker host group.
- Heavy Duty and High Availability
- In the Heavy Duty and High Availability templates, Kafka Connect roles are deployed in a dedicated host group named Connect. The instance count of this host group is set to zero by default. This means that, unless specifically provisioned, Kafka Connect roles are not deployed in Heavy Duty or High Availability clusters by default. The instance (node) count of the Connect host group can be configured in Data Hub during cluster provisioning. Additionally, you can add Kafka Connect to already running Heavy Duty or High Availability cluster by scaling the Connect host group.
Streams Messaging Manager integration
Kafka Connect deployed in CDP Public Cloud is by default configured to integrate with the Streams Messaging Manager (SMM) instance running in the cluster. This means that, if you provisioned your cluster with Kafka Connect, you will be able to manage, monitor, and deploy Kafka Connect connectors with SMM as soon as the cluster is deployed with no additional configuration required. Kafka Connect can be managed in SMM on the Connect page. For more information, see Monitoring Kafka Connect using Streams Messaging Manager.
