HDP2 to CDP Private Cloud Base Two Stage upgradePDF version

Ambari Metrics System (AMS) does not start

When the Ambari Metrics System (AMS) does not start after upgrade, you can observe the following log snippet in the HBase Master:

master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN,
                    ts=1543610616273, server=regionserver1.domain.com,41213,1543389145213}; ServerCrashProcedures=true.
                    Master startup cannot progress, in holding-pattern until region comes back online
                

The workaround is to manually clean up the Znode from ZooKeeper.

  • If AMS mode = embedded, Remove the Znode data from local filesystem path, for example:
     rm -f 
    /var/lib/ambari-metrics-collector/hbase-tmp/zookeeper/zookeeper_0/version-2/*
  • If AMS mode = distributed, connect to the cluster ZooKeeper instance and delete the following node before restart:
    /usr/hdp/current/zookeeper-client/bin/zkCli.sh -server localhost:2181
    [zk: localhost:2181(CONNECTED) 0] rmr /ams-hbase-unsecure/meta-region-server