Cloudera Docs

Scaling Kafka broker nodes

The Kafka broker nodes provisioned with the Light and Heavy duty versions of the Streams Messaging cluster definitions can be scaled. Additionally, if Cruise Control is available on the cluster, Cruise Control automatically rebalances partitions during a scaling operation. Before scaling Kafka nodes, Cloudera recommends that you review these notes regarding scaling operations.

Differences between newly provisioned and upgraded clusters

Clusters provisioned with Cloudera Runtime 7.2.12 or higher have the Cruise Control service deployed on them. Clusters that are upgraded from a previous version to 7.2.12 or higher do not have the Cruise Control service deployed on them. This is because during an upgrade, the services available on a cluster are not changed, Cruise Control is not added. Because the Cruise Control service is required for both downscale operations and automatic partition rebalancing, there are significant differences between what scaling features and operations are available for newly provisioned and upgraded clusters.
Clusters newly provisioned with 7.2.12 or higher
  • Support up and downscale operations.
  • Support automatic partition rebalancing with Cruise Control.
Clusters upgraded to 7.2.12 or higher
  • Support upscale operations only.
  • Partitions must be moved manually to any newly provisioned brokers after upscaling is finished.

Not all Kafka host groups are scalable

Streams Messaging clusters running Cloudera Runtime 7.2.12 or higher, have two host groups of Kafka broker nodes. These are the Core_broker and Broker host groups. During an upscale or downscale operation, new broker nodes are added to or removed from the Broker host group. The Core_broker group contains a core set of brokers and is not scalable. For more information regarding the Streams Messaging cluster templates, see Streams Messaging cluster layout.

Downscale operations ensure that the cluster remains healthy

During a downscale operation there are a number of automated safety checks to ensure that the cluster stays healthy at all times and no data is lost when brokers are decommissioned. For example:
  • Downscale operations are only carried out if sufficient resources are available. Otherwise they fail.
  • The partitions of a decommissioned broker are automatically moved to other brokers.
  • A broker is only decommissioned and removed from the broker's host group if there is no load on that broker.