Ambari-Metrics

To restore the backed-up Ambari-Metrics data, downgrade the ambari-metrics packages on all hosts and restore the backed-up-ambari-metrics data.

  1. Downgrade ambari-metrics packages on all hosts where they are installed.

    On every host in your cluster running a Metrics Monitor, run the following commands:

    yum downgrade ambari-metrics-monitor ambari-metrics-hadoop-sink
    Run the following command on all hosts running the Metrics Collector:
    yum downgrade ambari-metrics-collector
    Run the following command on the host running the Grafana component:
    yum downgrade ambari-metrics-grafana
  2. Restore backed-up ambari-metrics data on the host where the ambari-metrics-collector is installed
    rm -rf /var/lib/ambari-metrics-collector
    tar xf ams-backup.tar.gz  -C /
  3. Start Ambari-Metics in Ambari UI.
    Known Issue Description: AMS crashes after metrics DB rollback.
    Workaround:
    1. Stop Metrics Collector via ambari UI.
    2. Disable auto-start for Metrics Collector via ambari UI.
    3. SSH to metrics collector host.
    4. Find all processes run under AMS user and kill them.
    5. Remove PID files on the host (pid files are in /var/run/ambari-metrics-collector/ and /var/run/ams-hbase/ directories).
    6. Remove the previous content of hbase.rootdir and restore the back-up of metrics DB again.
    7. Remove the directory specified in AMS hbase.zookeeper.property.dataDir property.
    8. Remove the directory specified in AMS ams-hbase-site/phoenix.spool.directory property.
    9. Start Metrics Collector via ambari UI.
    10. Enable auto-start for Metrics Collector via ambari UI.