Deploying a NameNode HA Cluster

To deploy a NameNode HA cluster, you must initialize JournalNodes, run the required configurations for HA on the NameNodes, and validate the HA configuration.

In this task, we convert a non-HA cluster to HA. We use NN1 to denote the original NameNode in the non-HA setup, and NN2 to denote the other NameNode that is to be added in the HA setup. If you are using more than one standby NameNode, repeat the steps for NN2 on NN3.

Note

HA clusters reuse the NameService ID to identify a single HDFS instance (that may consist of multiple HA NameNodes).

A new abstraction called NameNode ID is added with HA. Each NameNode in the cluster has a distinct NameNode ID to distinguish it.

To support a single configuration file for all of the NameNodes, the relevant configuration parameters are suffixed with both the NameService ID and the NameNode ID.

Start the JournalNode daemons on those set of machines where the JNs are deployed. On each machine, execute the following command:
```
su –l hdfs –c "/usr/hdp/current/hadoop-hdfs-journalnode/../hadoop/sbin/hdfs-daemon.sh start journalnode"
```
Wait for the daemon to start on each of the JN machines.
Initialize JournalNodes.
- At the NN1 host machine, execute the following command:
```
su –l hdfs –c "hdfs namenode -initializeSharedEdits -force"
```
  This command formats all the JournalNodes. This by default happens in an interactive way: the command prompts users for “Y/N” input to confirm the format. You can skip the prompt by using option -force or -nonInteractive.
  
  It also copies all the edits data after the most recent checkpoint from the edits directories of the local NameNode (NN1) to JournalNodes.
- Initialize HA state in ZooKeeper. Execute the following command on NN1:
```
hdfs zkfc -formatZK -force
```
  This command creates a znode in ZooKeeper. The failover system stores uses this znode for data storage.
- Check to see if ZooKeeper is running. If not, start ZooKeeper by executing the following command on the ZooKeeper host machine(s).
```
su - zookeeper -c "export ZOOCFGDIR=/usr/hdp/current/zookeeper-server/conf ; export ZOOCFG=zoo.cfg; source /usr/hdp/current/zookeeper-server/conf/zookeeper-env.sh ; /usr/hdp/current/zookeeper-server/bin/zkServer.sh start"
```
- At the standby namenode host, execute the following command:
```
su -l hdfs -c "hdfs namenode -bootstrapStandby -force"
```
Start NN1. At the NN1 host machine, execute the following command:
```
su -l hdfs -c "/usr/hdp/current/hadoop-hdfs-namenode/../hadoop/sbin/hdfs-daemon.sh start namenode"
```
Make sure that NN1 is running correctly.
Format NN2 and copy the latest checkpoint (FSImage) from NN1 to NN2 by executing the following command:
```
su -l hdfs -c "hdfs namenode -bootstrapStandby -force"
```
This command connects with HH1 to get the namespace metadata and the checkpointed fsimage. This command also ensures that NN2 receives sufficient editlogs from the JournalNodes (corresponding to the fsimage). This command fails if JournalNodes are not correctly initialized and cannot provide the required editlogs.
Start NN2. Execute the following command on the NN2 host machine:
```
su -l hdfs -c "/usr/hdp/current/hadoop-hdfs-namenode/../hadoop/sbin/hadoop-daemon.sh start namenode"
```
Ensure that NN2 is running correctly.
If you are deploying an additional NameNode NN3, repeat steps 5 and 6 for NN3.

Start DataNodes. Execute the following command on all the DataNodes:

su -l hdfs -c "/usr/hdp/current/hadoop-hdfs-datanode/../hadoop/sbin/hadoop-daemon.sh start datanode"

Validate the HA configuration.

Go to the NameNodes' web pages separately by browsing to their configured HTTP addresses. Under the configured address label, you should see that HA state of the NameNode. The NameNode can be either in "standby" or "active" state.

	Note
	The HA NameNode is initially in the Standby state after it is bootstrapped. You can also use either JMX (tag.HAState) to query the HA state of a NameNode. The following command can also be used to query the HA state of a NameNode: `hdfs haadmin -getServiceState`

Transition one of the HA NameNodes to the Active state.
Initially, all the NameNodes are in the Standby state. Therefore you must transition one of the NameNodes to the Active state. This transition can be performed using one of the following options (ZooKeeper is not required for Option I).
- Option I: Using CLI – Use the command line interface (CLI) to transition one of the NameNode to the Active State. Execute the following command on that NameNode host machine:
```
hdfs haadmin -failover --forcefence --forceactive <serviceId> <namenodeId>
```
- Option II: Deploying Automatic Failover – You can configure and deploy automatic failover.