Ambari Metrics System (AMS) does not start

When the Ambari Metrics System (AMS) does not start after upgrade, you can observe the following log snippet in the HBase Master:

master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN,
                    ts=1543610616273, server=regionserver1.domain.com,41213,1543389145213}; ServerCrashProcedures=true.
                    Master startup cannot progress, in holding-pattern until region comes back online
                

The workaround is to manually clean up the Znode from ZooKeeper.

  • If AMS mode = embedded, Remove the Znode data from local filesystem path, for example:
     rm -f 
    /var/lib/ambari-metrics-collector/hbase-tmp/zookeeper/zookeeper_0/version-2/*
  • If AMS mode = distributed, connect to the cluster ZooKeeper instance and delete the following node before restart:
    /usr/hdp/current/zookeeper-client/bin/zkCli.sh -server localhost:2181
    [zk: localhost:2181(CONNECTED) 0] rmr /ams-hbase-unsecure/meta-region-server