Enabling High Availability and automatic failover

You can use Cloudera Manager to configure your CDP cluster for HDFS HA and automatic failover. In Cloudera Manager, HA is implemented using Quorum-based storage. Quorum-based storage relies upon a set of JournalNodes, each of which maintains a local edits directory that logs the modifications to the namespace metadata. Enabling HA enables automatic failover as part of the same command.

Cluster Administrator (also provided by Full Administrator)
The Enable High Availability workflow leads you through adding a second (standby) NameNode and configuring JournalNodes.

Go to the HDFS service.
Select Actions > Enable High Availability.
A screen showing the hosts that are eligible to run a standby NameNode and the JournalNodes displays.
1. Specify a name for the nameservice and click Continue.
  
  note
  Use unique names for the nameservices.
2. In the NameNode Hosts field, click Select a host.
  The host selection dialog box displays.
3. Select the checkbox next to the hosts where you want the standby NameNode to be set up and click OK.
  The standby NameNode cannot be on the same host as the active NameNode, and the host that is chosen should have the same hardware configuration (RAM, disk space, number of cores, and so on) as the active NameNode.
4. In the JournalNode Hosts field, click Select hosts.
  The host selection dialog box displays.
5. Check the checkboxes next to an odd number of hosts (a minimum of three) to act as JournalNodes and click OK.
  JournalNodes should be hosted on hosts with similar hardware specification as the NameNodes. Cloudera recommends that you put a JournalNode each on the same hosts as the active and standby NameNodes, and the third JournalNode on similar hardware, such as the JobTracker.
6. Click Continue.
7. In the JournalNode Edits Directory property, enter a directory location for the JournalNode edits directory into the fields for each JournalNode host.
  - You may enter only one directory for each JournalNode. The paths do not need to be the same on every JournalNode.
  - The directories you specify should be empty.
  - The directory owner should be hdfs:hadoop and must have read, write, and execute permission (drwx------).
8. Extra Options: Decide whether Cloudera Manager should clear existing data in ZooKeeper, standby NameNode, and JournalNodes.
  If the directories are not empty (for example, you are re-enabling a previous HA configuration), Cloudera Manager will not automatically delete the contents—you can select to delete the contents by keeping the default checkbox selection. The recommended default is to clear the directories. If you choose not to do so, the data should be in sync across the edits directories of the JournalNodes and should have the same version data as the NameNodes.
9. Click Continue.
  
  Cloudera Manager executes a set of commands that stop the dependent services, delete, create, and configure roles and directories as required, create a nameservice and failover controller, restart the dependent services, and deploy the new client configuration.
  
  important
  Some steps, such as formatting the NameNode may report failure if the action was already completed. However, the configuration steps continue to execute after reporting non-critical failed steps.
  
  note
  The HDFS Instances page denotes the active and standby NameNodes in parentheses next to the respective role instances.
Configure HDFS HA for other CDP services, if required.
Configuring other CDP components to use HDFS HA

important

If you change the NameNode Service RPC Port (dfs.namenode.servicerpc-address) while automatic failover is enabled, this will cause a mismatch between the NameNode address saved in the ZooKeeper /hadoop-ha znode and the NameNode address that the Failover Controller is configured with. This will prevent the Failover Controllers from restarting. If you need to change the NameNode Service RPC Port after Auto Failover has been enabled, you must do the following to re-initialize the znode:

Stop the HDFS service.
Configure the service RPC port:
1. Go to the HDFS service.
2. Click the Configuration tab
3. Select Scope > NameNode.
4. Select Category > Ports and Addresses.
5. Locate the NameNode Service RPC Port property or search for it by typing its name in the Search box.
6. Change the port value as needed.
  To apply this configuration property to other role groups as needed, edit the value for the appropriate role group.
On a ZooKeeper server host, run zookeeper-client.
1. Execute the following to remove the configured nameservice. This example assumes the name of the nameservice is nameservice1. You can identify the nameservice from the Federation and High Availability section on the HDFS Instances tab:
```
rmr /hadoop-ha/nameservice1
```
Click the Instances tab.
Select Actions > Initialize High Availability State in ZooKeeper.
Start the HDFS service.