Capacity estimates and goals

Learn about how to configure capacity estimations and goals for rebalancing with Cruise Control.

For the rebalancing, you need to provide the capacity values of your resources. These values are used for specifying the rebalancing criteria for your deployment. The following capacity values must be set:
Capacity Description
capacity.default.cpu 100 by default
capacity.default.network-in Given by the internet provider
capacity.default.network-out

The optimizers in Cruise Control use the network incoming and outgoing capacities to define a boundary for optimization. The capacity estimates are generated and read by Cruise Control. A capacity.json file is generated when Cruise Control is started. When a new broker is added, Cruise Control uses the default broker capacity values. However, in case disk related goals are used, Cruise Control must be restarted to load the actual disk capacity metrics of the new broker.

After setting the capacity estimates, you can provide different goals that define the optimization proposals given by Cruise Control. Default.goals are used to pre-compute optimization proposals that can be applied regardless of any anomalies. These default.goals on a healthy cluster can optimize resource utilization. Supported goals are also available to assist the optimized rebalancing process. When these goals are fulfilled, the rebalancing is successful. When the goals are violated, self-healing can be used and rebalancing must be carried out.

Cruise Control has an anomaly detection feature where goal violations can also be set. The anomaly.detection.goals configuration defines when the goals are not met, thus causing a violation. These anomalies can be fixed by the proposal generated from the self.healing.goals configuration. In case there is no self-healing goal specified, Cruise Control uses the default.goals setting. Hard goals can also be set to guarantee the fulfilment of any optimization or self-healing process. You can set the anomaly-detection goals, self-healing goals, and hard goals in Cloudera Manager under the Cruise Control Server Advanced Configuration Snippet (Safety Valve) for cruisecontrol.properties setting.

You can find the capacity estimate and goal configurations at Cloudera Manager>Cruise Control>Configuration>Main.

The following table lists all the configurations that are needed to configure Cruise Control specifically to your environment. For the list of goals, see the upstream Cruise Control documentation.

Configuration Description
num.metric.fetchers Parallel threads for fetching metrics from the Cloudera Manager database
default.goals List of default goals
goals List of subset goals
partition.metric.sample.store.topic Storing Cruise Control metrics
broker.metric.sample.store.topic Storing Cruise Control metircs
partition.metrics.window.ms Time window size for partition metrics
broker.metrics.window.ms Time window size for broker metrics
num.partition.metrics.windows Number of stored partition windows
num.broker.metrics.windows Number of stored broker windows