1. Enabling CGroups

CGroups is a Linux kernel feature. Currently HDP supports CGroups on RHEL6 only. At this time there is no CGroups equivalent for Windows. CGroups are not enabled by default on HDP. CGroups require that the HDP cluster be Kerberos enabled.

Enable CGroups

On Centos 6, CGroups are not set up by default. Run the following commands to set up CGroups:

yum install libcgroup
sudo mkdir /cgroup
mount -t cgroup -o cpu cpu /cgroup

Set the following properties in the /etc/hadoop/conf/yarn-site.xml file on the ResourceManager and NodeManager hosts:

Property: yarn.nodemanager.container-executor.class

Value: org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor

Example:

<property>
 <name>yarn.nodemanager.container-executor.class</name>
 <value>org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor</value>
</property>

Property: yarn.nodemanager.linux-container-executor.group

Value: hadoop

Example:

<property>
 <name>yarn.nodemanager.linux-container-executor.group</name>
 <value>hadoop</value>
</property>

Property: yarn.nodemanager.linux-container-executor.resources-handler.class

Value: org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler

Example:

<property>
 <name>yarn.nodemanager.linux-container-executor.resources-handler.class</name>
 <value>org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler</value>
</property>

Property: yarn.nodemanager.linux-container-executor.cgroups.hierarchy

Value: /yarn

Example:

<property>
 <name>yarn.nodemanager.linux-container-executor.cgroups.hierarchy</name>
 <value>/yarn</value>
</property>

Property: yarn.nodemanager.linux-container-executor.cgroups.mount

Value: true

Example:

<property>
 <name>yarn.nodemanager.linux-container-executor.cgroups.mount</name>
 <value>true</value>
</property>

Property: yarn.nodemanager.linux-container-executor.cgroups.mount-path

Value: /cgroup

Example:

<property>
 <name>yarn.nodemanager.linux-container-executor.cgroups.mount-path</name>
 <value>/cgroup</value>
</property>

Set the Percentage of CPU used by YARN

Set the percentage of CPU that can be allocated for YARN containers. In most cases, the default value of 100% should be used. If you have another process that needs to run on a node that also requires CPU resources, you can lower the percentage of CPU allocated to YARN to free up resources for the other process.

Property: yarn.nodemanager.resource.percentage-physical-cpu-limit

Value: 100

Example:

<property>
 <name>yarn.nodemanager.resource.percentage-physical-cpu-limit</name>
 <value>100</value>
</property>

Set Flexible or Strict CPU limits

CPU jobs are constrained with CPU scheduling and CGroups enabled, but by default these are flexible limits. If spare CPU cycles are available, containers are allowed to exceed the CPU limits set for them. With flexible limits, the amount of CPU resources available for containers to use can vary based on cluster usage -- the amount of CPU available in the cluster at any given time.

You can use CGroups to set strict limits on CPU usage. When strict limits are enabled, each process receives only the amount of CPU resources it requests. With strict limits, a CPU process will receive the same amount of cluster resources every time it runs.

Strict limits are not enabled (set to false) by default. To enable strict CPU limits, set the following property to true.

Property: yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage

Value: true

Example:

 <property>
 <name>yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage</name>
 <value>true</value>
 </property>
    

	Note
	Irrespective of whether this property is true or false, at no point will total container CPU usage exceed the limit set in yarn.nodemanager.resource.percentage-physical-cpu-limit.