Managing YARN

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

1. On the Home > Status tab, click to the right of the cluster name and select Add a Service. A list of service types display. You can add one type of service at a time.
2. Select YARN (MR2 Included) and click Continue.
3. Select the services on which the new service should depend. All services must depend on the same ZooKeeper service. Click Continue.
4. Customize the assignment of role instances to hosts. The wizard evaluates the hardware configurations of the hosts to determine the best hosts for each role. The wizard assigns all worker roles to the same set of hosts to which the HDFS DataNode role is assigned. You can reassign role instances.

   Click a field below a role to display a dialog box containing a list of hosts. If you click a field containing multiple hosts, you can also select All Hosts to assign the role to all hosts, or Custom to display the hosts dialog box.

   The following shortcuts for specifying hostname patterns are supported:

   • Range of hostnames (without the domain portion)

     Range Definition	Matching Hosts
     10.1.1.[1-4]	10.1.1.1, 10.1.1.2, 10.1.1.3, 10.1.1.4
     host[1-3].company.com	host1.company.com, host2.company.com, host3.company.com
     host[07-10].company.com	host07.company.com, host08.company.com, host09.company.com, host10.company.com

   • IP addresses
   • Rack name

   Click the View By Host button for an overview of the role assignment by hostname ranges.

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

Minimum Required Role: Configurator (also provided by Cluster Administrator, Full Administrator)

The YARN service is configured by default to use the Fair Scheduler. You can change the scheduler type to FIFO or Capacity Scheduler. You can also modify the Fair Scheduler and Capacity Scheduler configuration. For further information on schedulers, see YARN (MRv2) and MapReduce (MRv1) Schedulers.

In addition to the static resource management available to all services, the YARN service also supports dynamic management of its static allocation. See Dynamic Resource Pools.

The Fair Scheduler does not support Dominant Resource Calculator. The fairshare policy that the Fair Scheduler uses considers only the memory for fairShare and minShare calculation, therefore GPU devices will be allocated from a common pool.

This section describes how to define a Resource Type other than memory, CPU or GPU. For instructions on how to enable GPU as a resource, see Enable GPU Using Cloudera Manager.

1. Select the YARN service in Cloudera Manager.
2. Click the Configuration tab.
3. Search for resource types.
4. Find Resource Types.
5. Click the plus icon on the right side to create a new Resource Type.
6. Add the following properties:
   • Name: Add the name of the resource type. For more information about valid resource name, see YARN Resource Configuration.
   • Description: Add description of the resource type. Optional property.
   • Minimum value: Define the minimum allocation size of the resource type for a container. Its default value is 0.
   • Maximum value: Define the maximum allocation size of the resource type for a container. Its default value is Long.MAX_VALUE.
   • Unit: Define the default unit of the resource type.
7. Find Resource Allocation.
8. Allocate the created resource type to a Role Group by clicking the plus icon on the right side of the row of the particular Role Group.
9. Add value to define the size of the allocated resource to that Role Group.
10. Click Save Changes.
11. If you want to constraint the queues for the defined Resource Type, do the followings:
    1. Select the Hosts / All hosts tab.
    2. Find the the host with the ResourceManager role and then click on the ResourceManager role.
    3. Click the Process tab.
    4. Open the fair-scheduler.xml and copy its content.
    5. Return to the Home page by clicking the Cloudera Manager logo.
    6. Select the YARN service.
    7. Click the Configuration tab.
    8. Search for fair scheduler.
    9. Find Fair Scheduler XML Advanced Configuration Snippet (Safety Valves).
    10. Paste the content you copied from the fair-scheduler.xml.
    11. Specify the custom resources using the vcores=X, memory-mb=Y format.

        For more information about the Allocation file format, see YARN documentation on FairScheduler.

12. Click Save Changes.
13. Click the Stale Service Restart icon that is next to the service to invoke the cluster restart wizard.

    As a result the changes in the stale configuration are displays.

    In yarn-conf/yarn-site.xml:

    <name>yarn.resource-types</name>
    <value>fpga</value>
    </property>
    <property>
    <name>yarn.resource-types.fpga.minimum-allocation</name>
    <value>0</value>
    </property>
    <property>
    <name>yarn.resource-types.fpga.maximum-allocation</name>
    <value>10</value>
    </property>
    <property>

    In yarn-site.xml:

    <name>yarn.resource-types</name>
    <value>fpga</value>
    </property>
    <property>
    <name>yarn.resource-types.fpga.minimum-allocation</name>
    <value>0</value>
    </property>
    <property>
    <name>yarn.resource-types.fpga.maximum-allocation</name>
    <value>10</value>
    </property>
    <property>

    <name>yarn.resource-types</name>
    <value>fpga</value>
    </property>
    <property>
    <name>yarn.nodemanager.resource-type.fpga</name>
    <value>2</value>
    </property>
    <property>

14. Click Restart Stale Services.
15. Select the Re-deploy client configuration.
16. Click Restart Now.

On a secure cluster, long-running applications such as Spark Streaming jobs will need additional configuration since the default settings only allow the hdfs user's delegation tokens a maximum lifetime of 7 days, which is not always sufficient.

All YARN tasks on the NodeManager are run in a JVM. When a task runs successfully, the exit code is 0. Exit codes of 0 are not logged, as they are the expected result. Any non-zero exit code is logged as an error. The non-zero exit code is reported by the NodeManager as an error in the child process. The NodeManager itself is not affected by the error.