How Cruise Control rebalancing works
You can use Cruise Control as a load balancing component in large Kafka installations to automatically balance the partitions based on specific conditions for your deployment. The elements in the Cruise Control architecture are responsible for different parts of the rebalancing process that uses Kafka metrics and optimization goals.
Generates a cluster workload model based on standard Kafka metrics and resource metrics to utilize disk, CPU, bytes-in rate and bytes-out rate. Feeds the cluster model into Anomaly Detector and Analyzer.
Generates optimization proposals based on optimization goals provided by the user, and cluster workload model from Load Monitor. Hard goals and soft goals can be set. Hard goals must be fulfilled, while soft goals can be left unfulfilled if hard goals are reached. The optimization fails if the hard goal is violated by optimization results.
|Broker failure||Non-empty broker crashes or leaves a cluster.||Cruise Control fixes the cluster by removing the failed brokers.|
|Goal violations||Optimization is violated.||Cruise Control automatically analyzes the workload and executes the optimization proposals, if self-healing is enabled.|
|Disk failure||Non-empty disk dies.||Cruise Control moves all the offline replicas to healthy brokers, if self-healing is enabled.|
|Metric anomaly||Anomalies in the collected metrics.||May first demote, and if the anomaly persists, remove slow brokers depending on the
Carries out the optimization proposals and it can be safely interrupted when executing proposals. The executions are always resource-aware processes.