Resolving Cluster Upgrade Problems
Try the recommended solution for each of the following problems.
Problem: Versions tab does not show in Ambari Web.
After performing an upgrade from HDP 2.1 and restarting Ambari Server and the Agents, if you browse to Admin > Stack and Versions in Ambari Web, the Versions tab does not display.
Solution:
Give all the Agent hosts in the cluster a chance connect to Ambari Server by waiting for Ambari to show the Agent heartbeats as green and then refresh your browser.
Problem: YARN Service Checks Fail and ResourceManager fails to start
When upgrading from HDP 2.2 -> 2.3, if your cluster contains
yarn.scheduler.capacity.root.accessible-node-labels.default.capacity
and
yarn.scheduler.capacity.root.accessible-node-labels.default.maximum-capacity
properties, these values must be valid before upgrading to HDP 2.3. During HDP upgrade, on
ResourceManager start, if these values are invalid, you will get the following error:
Illegal capacity of 0.0 for children of queue root for label=defaultUpdate YARN Configuration Properties for HDP 2.3
Solution:
From Ambari Web, browse to Services > YARN > Configs. On the Advanced tab, delete the following properties from capacity-scheduler:
yarn.scheduler.capacity.root.accessible-node-labels.default.capacity
yarn.scheduler.capacity.root.accessible-node-labels.default.maximum-capacity
Also, be sure these properties are valid (or not included) in your Blueprints when you create clusters.
Problem: After HDP 2.2 -> 2.3 Manual Upgrade, Ambari alerts service flags Kafka Broker down.
After performing a manual upgrade from HDP 2.2 - 2.3, Ambari display alerts for Kafka Brokers.
Solution:
During upgrade, Ambari adds the listeners=PLAINTEXT://localhost:6667 to /etc/kafka/conf/server.properties this cause Kafka to listen on localhost at port 6667 and Ambari alerts service will not be able to reach Kafka broker and flags it as service down.users can run following command. It is not until Ambari set-current is run (per the upgrade instructions) that this value is corrected.