Enabling self-healing for all or individual anomaly types

Self-healing is disabled for Cruise Control by default. You can enable self-healing in Cloudera Manager using the cruisecontrol.properties configuration, or with a curl POST request and the corresponding anomaly type.

Enabling self-healing in Cloudera Manager

  1. Go to your cluster in Cloudera Manager.
  2. Select Cruise Control from the list of Services.
  3. Click on Configuration tab.
  4. Search for the Cruise Control Server Advanced Configuration Snippet (Safety Valve) for cruisecontrol.properties setting.
  5. Choose to enable self-healing for all or only specific anomaly types, and add the corresponding parameter to the Safety Valve field based on your requirements.
    For all anomaly types For specific anomaly types
    self.healing.enabled=true self.healing.broker.failure.enabled=true
    self.healing.goal.violation.enabled=true
    self.healing.disk.failure.enabled=true
    self.healing.topic.anomaly.enabled=true
    self.healing.slow.broker.removal.enabled=true
  6. Click Save changes.
  7. Click on Action > Restart next to the Cruise Control service name to restart Cruise Control.

Enabling self-healing using REST API

  1. Open a command line tool.
  2. Use ssh and connect to your cluster running Cruise Control.
    ssh root@<your_hostname>

    You will be prompted to provide your password.

  3. Enable self-healing for the required anomaly types using the following POST command:
    POST /kafkacruisecontrol/admin?enable_self_healing_for=[anomaly_type]
    The following parameters must be used for anomaly_type:
    • GOAL_VIOLATION
    • BROKER_FAILURE
    • METRIC_ANOMALY
    • DISK_FAILURE
    • TOPIC_ANOMALY
  4. Check which anomalies are currently in use, and which are detected with the following GET command:
    GET /kafkacruisecontrol/state
When reviewing the state of Cruise Control, you can check the status of Anomaly Detector at the following parameters:
  • selfHealingEnabled - Anomaly type for which self-healing is enabled
  • selfHealingDisabled - Anomaly type for which self healing is disabled
  • recentGoalViolations - Recently detected goal violations
  • recentBrokerFailures - Recently detected broker failures
  • recentDiskFailures - Recently detected disk failures
  • recentMetricAnomalies - Recently detected metric anomalies