How Cruise Control rebalancing works

You can use Cruise Control as a load balancing component in large Kafka installations to automatically balance the partitions based on specific conditions for your deployment. The elements in the Cruise Control architecture are responsible for different parts of the rebalancing process that uses Kafka metrics and optimization goals.

The following illustration shows the architecture of Cruise Control.

Load Monitor

Generates a cluster workload model based on standard Kafka metrics and resource metrics to utilize disk, CPU, bytes-in rate and bytes-out rate. Feeds the cluster model into Anomaly Detector and Analyzer.

Analyzer

Generates optimization proposals based on optimization goals provided by the user, and cluster workload model from Load Monitor. Hard goals and soft goals can be set. Hard goals must be fulfilled, while soft goals can be left unfulfilled if hard goals are reached. The optimization fails if the hard goal is violated by optimization results.

Anomaly Detector

Responsible for detecting the following anomalies:


Anomaly	Cause	Result
Broker failure	Non-empty broker crashes or leaves a cluster.	Cruise Control fixes the cluster by removing the failed brokers.
Goal violations	Optimization is violated.	Cruise Control automatically analyzes the workload and executes the optimization proposals, if self-healing is enabled.
Disk failure	Non-empty disk dies.	Cruise Control moves all the offline replicas to healthy brokers, if self-healing is enabled.
Metric anomaly	Anomalies in the collected metrics.	May first demote, and if the anomaly persists, remove slow brokers depending on the `self.healing.slow.brokers.removal.enabled` configuration.

Executor

Carries out the optimization proposals and it can be safely interrupted when executing proposals. The executions are always resource-aware processes.