Apache Ambari Operations
Also available as:
PDF
loading table of contents...

Configuring NameNode High Availability

Prerequisites

  • Verify that you have at least three hosts in your cluster and are running at least three Apache ZooKeeper servers.

  • Verify that the Hadoop Distributed File System (HDFS) and ZooKeeper services are not in Maintenance Mode.

    HDFS and ZooKeeper must stop and start when enabling NameNode HA. Maintenance Mode will prevent those start and stop operations from occurring. If the HDFS or ZooKeeper services are in Maintenance Mode the NameNode HA wizard will not complete successfully.

Steps

  1. In Ambari Web, select Services > HDFS > Summary.

  2. Click Service Actions, then click Enable NameNode HA.

  3. The Enable HA wizard launches. This wizard describes the set of automated and manual steps you must take to set up NameNode high availability.

  4. On the Get Started page, type in a Nameservice ID and click Next.

    You use this Nameservice ID instead of the NameNode FQDN after HA is set up.

  5. On the Select Hosts page, select a host for the additional NameNode and the JournalNodes, and then click Next:

  6. On the Review page, confirm your host selections and click Next:

  7. Follow the directions on the Manual Steps Required: Create Checkpoint on NameNode page, and then click Next:

    You must log in to your current NameNode host and run the commands to put your NameNode into safe mode and create a checkpoint.

  8. When Ambari detects success and the message on the bottom of the window changes to Checkpoint created, click Next.

  9. On the Configure Components page, monitor the configuration progress bars, then click Next:

  10. Follow the instructions on the Manual Steps Required: Initialize JournalNodes page and then click Next:

    You must log in to your current NameNode host to run the command to initialize the JournalNodes.

  11. When Ambari detects success and the message on the bottom of the window changes to JournalNodes initialized, click Next.

  12. On the Start Components page, monitor the progress bars as the ZooKeeper servers and NameNode start; then click Next:

    [Note]Note

    In a cluster with Ranger enabled, and with Hive configured to use MySQL, Ranger will fail to start if MySQL is stopped. To work around this issue, start the Hive MySQL database and then retry starting components.

  13. On the Manual Steps Required: Initialize NameNode HA Metadata page : Complete each step, using the instructions on the page, and then click Next.

    For this step, you must log in to both the current NameNode and the additional NameNode. Make sure you are logged in to the correct host for each command. Click OK to confirm, after you complete each command.

  14. On the Finalize HA Setup page, monitor the progress bars as the wizard completes HA setup, then click Done to finish the wizard.

    After the Ambari Web UI reloads, you may see some alert notifications. Wait a few minutes until all the services restart.

  15. Restart any components using Ambari Web, if necessary.

  16. If you are using Hive, you must manually change the Hive Metastore FS root to point to the Nameservice URI instead of the NameNode URI. You created the Nameservice ID in the Get Started step.

    Steps

    1. Find the current FS root on the Hive host:

      hive --config /etc/hive/conf/conf.server --service metatool -listFSRoot

      The output should look similar to Listing FS Roots... hdfs://<namenode-host>/apps/hive/warehouse.

    2. Change the FS root:

      $ hive --config /etc/hive/conf/conf.server --service metatool -updateLocation <new-location><old-location>

      For example, if your Nameservice ID is mycluster, you input:

      $ hive --config /etc/hive/conf/conf.server --service metatool -updateLocation hdfs://mycluster/apps/hive/warehouse hdfs://c6401.ambari.apache.org/apps/hive/warehouse.

      The output looks similar to:

      Successfully updated the following locations...Updated X records in SDS table

      [Important]Important

      The Hive configuration path for a default HDP 2.3.x or later stack is /etc/hive/conf/conf.server

      The Hive configuration path for a default HDP 2.2.x or earlier stack is /etc/hive/conf

  17. Adjust the ZooKeeper Failover Controller retries setting for your environment:

    1. Browse to Services > HDFS > Configs > Advanced core-site.

    2. Set ha.failover-controller.active-standby-elector.zk.op.retries=120.

Next Steps

Review and confirm all recommended configuration changes.

More Information

Review and Confirm Configuration Changes