Work Preserving Recovery for YARN Components

CDH 5.2 introduces work preserving recovery for the YARN ResourceManager and NodeManager. With work preserving recovery enabled, if a ResourceManager or NodeManager restarts, no in-flight work is lost. You can configure work preserving recovery separately for a ResourceManager or NodeManager.

Prerequisites

To use work preserving recovery for the ResourceManager, you need to first enable High Availability for the ResourceManager. See YARN (MRv2) ResourceManager High Availability for more information.

Configuring Work Preserving Recovery Using Cloudera Manager

Enabling Work Preserving Recovery on ResourceManager with Cloudera Manager

If you use Cloudera Manager and you enable YARN (MRv2) ResourceManager High Availability, work preserving recovery is enabled by default for the ResourceManager.

Disabling Work Preserving Recovery on ResourceManager with Cloudera Manager

To disable the feature for the ResourceManager, change the value of yarn.resourcemanager.work-preserving-recovery.enabled to false in the yarn-site.xml using an advanced configuration snippet.

  1. In Cloudera Manager>Home>Status, click the yarn link.
  2. Click the Configuration tab.
  3. Search for ResourceManager yarn-site.xml.
  4. In the ResourceManager Base Group field, enter the following XML element, setting the feature to false. 
    <property>
  <name>yarn.resourcemanager.work-preserving-recovery.enabled</name>
  <value>false</value>
</property>
  5. Click Save Changes.

Enabling Work Preserving Recovery on NodeManager with Cloudera Manager

The default value for the recovery directory is ${hadoop.tmp.dir}/yarn-nm-recovery. This location usually points to the /tmp directory on the local filesystem. Because many operating systems do not preserve the contents of the /tmp directory across a reboot, Cloudera strongly recommends that you change the location of yarn.nodemanager.recover.dir to a different directory on the local filesystem. The example below uses /home/cloudera/recovery.

To enable work preserving recovery for a given NodeManager:
  1. Edit the advanced configuration snippet for yarn-site.xml on that NodeManager, and set the value of yarn.nodemanager.recovery.enabled to true.
  2. Configure the directory on the local filesystem where state information is stored when work preserving recovery is enabled.
    1. In Cloudera Manager Home, click the yarn link.
    2. Click the Configuration tab.
    3. Search for yarn.nodemanager.recovery.dir.
    4. Enter the directory path in the NodeManager Base Group field (for example, /home/cloudera/recovery).
    5. Click Save Changes.

Configuring Work Preserving Recovery Using the Command Line

After enabling YARN (MRv2) ResourceManager High Availability, add the following property elements to yarn-site.xml on the ResourceManager and all NodeManagers.
  1. Set the value of yarn.resourcemanager.work-preserving-recovery.enabled to true to enable work preserving recovery for the ResourceManager, and set the value of yarn.nodemanager.recovery.enabled to true for the NodeManager.
  2. For each NodeManager, configure the directory on the local filesystem where state information is stored when work preserving recovery is enabled, Set yarn.nodemanager.recovery.dir to a local filesystem directory. The default value is ${hadoop.tmp.dir}/yarn-nm-recovery. This location usually points to the /tmp directory on the local filesystem. Because many operating systems do not preserve the contents of the /tmp directory across a reboot, Cloudera strongly recommends changing the location of yarn.nodemanager.recovery.dir to a directory under the root partition. If the drive which hosts this directory fails, the NodeManager will also fail. The example below uses /home/cloudera/recovery.
  3. Configure a valid RPC address for the NodeManager by setting yarn.nodemanager.address to an address with a specific port number (such as 0.0.0.0:45454). Ephemeral ports (default is port 0) cannot be used for the NodeManager's RPC server; this could cause the NodeManager to use different ports before and after a restart, preventing clients from connecting to the NodeManager. The NodeManager RPC address is also important for auxiliary services that run in a YARN cluster.

Auxiliary services should be designed to support recoverability by reloading the previous state after a NodeManager restarts. An example auxiliary service, the ShuffleHandler service for MapReduce, follows the correct pattern for an auxiliary service that supports work preserving recovery of the NodeManager.

Example Configuration for Work Preserving Recovery

The following example configuration can be used with a Cloudera Manager advanced configuration snippet or added to yarn-site.xml directly if you do not use Cloudera Manager. Adjust the configuration to suit your environment. 
<property>
  <name>yarn.resourcemanager.work-preserving-recovery.enabled</name>
  <value>true</value>
  <description>Whether to enable work preserving recovery for the Resource Manager</description>
</property>
<property>
  <name>yarn.nodemanager.recovery.enabled</name>
  <value>true</value>
  <description>Whether to enable work preserving recovery for the Node Manager</description>
</property>
<property>
  <name>yarn.nodemanager.recovery.dir</name>
  <value>/home/cloudera/recovery</value>
  <description>The location for stored state on the Node Manager, if work preserving recovery 
    is enabled.</description>
</property>
<property>
  <name>yarn.nodemanager.address</name>
  <value>0.0.0.0:45454</value>
</property>