Scaling Kafka brokers
The Kafka brokers provisioned with the Light and Heavy duty versions of the Streams Messaging cluster definitions can be scaled. This is done by adding or removing nodes from the host groups containing Kafka brokers. During a scaling operation Cruise Control automatically rebalances partitions on the cluster. Before scaling Kafka brokers, Cloudera recommends that you review these notes regarding scaling operations.
Version limitations
- Clusters newly provisioned with Cloudera Runtime 7.2.12 or higher.
- Clusters upgraded from Cloudera Runtime 7.2.12 or higher to a newer version.
Not all Kafka host groups are scalable
Streams Messaging clusters running Cloudera Runtime 7.2.12 or higher have two host groups of Kafka broker nodes. These are the Core_broker and Broker host groups. During an upscale or downscale operation, new broker nodes are added to or removed from the Broker host group. The Core_broker group contains a core set of brokers and is not scalable. For more information regarding the Streams Messaging cluster templates, see Streams Messaging cluster layout.
Downscale operations ensure that the cluster remains healthy
- Downscale operations are only carried out if sufficient resources are available. Otherwise they fail.
- The partitions of a decommissioned broker are automatically moved to other brokers.
- A broker is only decommissioned and removed from the broker's host group if there is no load on that broker.