Scaling Kafka brokers

The Kafka brokers provisioned with the Light and Heavy duty versions of the Streams Messaging cluster definitions can be scaled. This is done by adding or removing nodes from the host groups containing Kafka brokers. During a scaling operation Cruise Control automatically rebalances partitions on the cluster. Before scaling Kafka brokers, Cloudera recommends that you review these notes regarding scaling operations.

Version limitations

Scaling brokers is only supported on:

Clusters newly provisioned with Cloudera Runtime 7.2.12 or higher.
Clusters upgraded from Cloudera Runtime 7.2.12 or higher to a newer version.

Not all Kafka host groups are scalable

Streams Messaging clusters running Cloudera Runtime 7.2.12 or higher have two host groups of Kafka broker nodes. These are the Core_broker and Broker host groups. During an upscale or downscale operation, new broker nodes are added to or removed from the Broker host group. The Core_broker group contains a core set of brokers and is not scalable. For more information regarding the Streams Messaging cluster templates, see Streams Messaging cluster layout.

Downscale operations ensure that the cluster remains healthy

During a downscale operation there are a number of automated safety checks to ensure that the cluster stays healthy at all times and no data is lost when brokers are decommissioned. For example:

Downscale operations are only carried out if sufficient resources are available. Otherwise they fail.
The partitions of a decommissioned broker are automatically moved to other brokers.
A broker is only decommissioned and removed from the broker's host group if there is no load on that broker.