Cloudera Docs

Scaling up Kafka broker nodes

Learn how you can upscale Kafka broker nodes in a Streams Messaging cluster.

The Kafka broker nodes in a Streams Messaging cluster can be upscaled. During an upscale operation, broker nodes are added to the Broker host group.

An upscale operation behaves differently depending on whether the Cruise Control service is available on the cluster. If Cruise Control is available on the cluster, newly commissioned brokers are automatically detected, a rebalance is triggered once they become available, and data is moved to the new brokers. If Cruise Control is not available on the cluster, data must be moved to the new brokers manually.

  • Ensure that you have reviewed Scaling Kafka broker nodes.
  • Ensure that the cluster, its hosts, and all its services are healthy.
  • Ensure that the Kafka brokers are commissioned and running.
  • Ensure that the Cruise Control service (if available) is running.
  1. Log in to the CDP web interface.
  2. Go to Management Console > Data Hub Clusters.
  3. Find and select the Data Hub cluster you want to scale.
  4. Click Actions > Resize.
  5. Select the Broker host group.
  6. Set the desired size of the host group.
    You can enter a number or use the plus and minus buttons to adjust the number of nodes.
  7. Click Resize.
  8. Wait until the new nodes become available.
  9. Access the Cloudera Manager instance managing the cluster.
  10. Restart all services that have stale configurations.
The cluster is upscaled. The number of Kafka broker nodes available in the Broker host group is equal to the configured number of nodes. If Cruise Control is available on the cluster, data is automatically moved to the new brokers.
If Cruise Control is not available on the cluster, move your data manually.