Scaling Kafka broker nodes
The Kafka broker nodes provisioned with the Light and Heavy duty versions of the Streams Messaging cluster definitions can be scaled. Additionally, if Cruise Control is available on the cluster, Cruise Control automatically rebalances partitions during a scaling operation. Before scaling Kafka nodes, Cloudera recommends that you review these notes regarding scaling operations.
Differences between newly provisioned and upgraded clusters
- Clusters newly provisioned with 7.2.12 or higher
- Support up and downscale operations.
- Support automatic partition rebalancing with Cruise Control.
- Clusters upgraded to 7.2.12 or higher
- Support upscale operations only.
- Partitions must be moved manually to any newly provisioned brokers after upscaling is finished.
Not all Kafka host groups are scalable
Streams Messaging clusters running Cloudera Runtime 7.2.12 or higher, have two host groups of Kafka broker nodes. These are the Core_broker and Broker host groups. During an upscale or downscale operation, new broker nodes are added to or removed from the Broker host group. The Core_broker group contains a core set of brokers and is not scalable. For more information regarding the Streams Messaging cluster templates, see Streams Messaging cluster layout.
Downscale operations ensure that the cluster remains healthy
- Downscale operations are only carried out if sufficient resources are available. Otherwise they fail.
- The partitions of a decommissioned broker are automatically moved to other brokers.
- A broker is only decommissioned and removed from the broker's host group if there is no load on that broker.