Ambari Metrics System (AMS) does not start
When the Ambari Metrics System (AMS) does not start after upgrade, you can observe the following log snippet in the HBase Master:
master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN,
ts=1543610616273, server=regionserver1.domain.com,41213,1543389145213}; ServerCrashProcedures=true.
Master startup cannot progress, in holding-pattern until region comes back online
The workaround is to manually clean up the Znode from ZooKeeper.
-
If AMS mode = embedded, Remove the Znode data from local filesystem path, for example:
rm -f /var/lib/ambari-metrics-collector/hbase-tmp/zookeeper/zookeeper_0/version-2/*
- If AMS mode = distributed, connect to the cluster ZooKeeper instance and
delete the following node before restart:
/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server localhost:2181 [zk: localhost:2181(CONNECTED) 0] rmr /ams-hbase-unsecure/meta-region-server