Scaling up Kafka Connect

Learn how you can upscale Kafka Connect in a Streams Messaging cluster.

Kafka Connect in a Streams Messaging cluster can be upscaled. The upscale operation differs depending on which Streams Messaging cluster template you are using. In the Light Duty template Kafka Connect is upscaled by adding nodes to the Broker host group. In the Heavy Duty template, Kafka Connect is upscaled by adding nodes to the Connect host group.

Ensure that you have reviewed Scaling Kafka Connect.
Ensure that the cluster, its hosts, and all its services are healthy.
Ensure that the Kafka brokers are commissioned and running.
If using the Light Duty cluster template, ensure that the Cruise Control service is running.
This is required because in the Light Duty template, Kafka Connect is deployed on the same node as Kafka brokers. While Kafka Connect does not require Cruise Control for scaling, Kafka brokers do.

Log in to the CDP web interface.
Go to Management Console > Data Hub Clusters.
Find and select the Data Hub cluster you want to scale.
Click Actions > Resize.
Select the host group you want to scale.
For Light Duty clusters, select the Broker host group. For Heavy Duty clusters, select the Connect host group.
Set the desired size of the host group.
You can enter a number or use the plus and minus buttons to adjust the number of nodes.
Click Resize.
Wait until the new nodes become available.
Access the Cloudera Manager instance managing the cluster.
Restart all services that have stale configurations.

The cluster is upscaled. In Light Duty clusters, the number of nodes available in the Broker host group is equal to the configured number of nodes. In Heavy Duty clusters, the number of nodes available in the Connect host group is equal to the configured number of nodes.