Scaling up Kafka brokers

Learn how you can upscale Kafka brokers in a Streams Messaging cluster.

The Kafka brokers in a Streams Messaging cluster can be upscaled. During an upscale operation, broker nodes are added to the Broker host group. Newly commissioned brokers are automatically detected by Cruise Control, a rebalance is triggered once they become available, and data is moved to the new brokers.

Ensure that you have reviewed Scaling Kafka broker nodes.
Ensure that the cluster, its hosts, and all its services are healthy.
Ensure that the Kafka brokers are commissioned and running.
Ensure that the Cruise Control service is running.

Log in to the CDP web interface.
Go to Management Console > Data Hub Clusters.
Find and select the Data Hub cluster you want to scale.
Click Actions > Resize.
Select the Broker host group.
Set the desired size of the host group.
You can enter a number or use the plus and minus buttons to adjust the number of nodes.
Click Resize.
Wait until the new nodes become available.
Access the Cloudera Manager instance managing the cluster.
Restart all services that have stale configurations.

The cluster is upscaled. The number of Kafka broker nodes available in the Broker host group is equal to the configured number of nodes. Data is automatically moved to the new brokers by Cruise Control.