Scaling Kafka Connect

Kafka Connect provisioned with the Light and Heavy duty versions of the Streams Messaging cluster definitions can be scaled. However, there are differences between how scaling is performed in each of the templates. Cloudera recommends that you review the following notes and recommendations before you scale Kafka Connect.

Light Duty

In the Light duty cluster template, Kafka Connect roles are deployed in the Broker and Core_broker host groups. The Core_boker host group has a default instance count of 3, the Broker host group has a default instance count of 0. This means that by default all Light Duty clusters will a have a minimum of three Kafka Connect roles. To scale Kafka Connect in a Light Duty cluster, you resize the Broker host group. The Core_broker host group is not scalable. When scaling the Broker host group all other service roles are also scaled. Scaling Kafka Connect on its own in Light Duty clusters is not possible.

Heavy Duty

In the Heavy Duty cluster template, Kafka Connect roles are deployed in a dedicated host group named Connect. To scale Kafka Connect in a Heavy Duty cluster, you resize the Connect host group. The instance count of this host group is set to 0 by default. This means that unless specifically provisioned, no Kafka Connect roles are deployed in Heavy Duty clusters.

For more information regarding the Streams Messaging cluster templates, see Streams Messaging cluster layout.