Enable Cgroups

You can enable CPU Scheduling to enable cgroups. You must configure certain properties in yarn-site.xml on the ResourceManager and NodeManager hosts to enable cgroups.

Follow these steps to enable Cgroup-based resource management at Host level configuration on Cloudera Manager:
  1. Navigate to the Hosts option in the left navigation pane.
  2. Select Hosts Configuration. The Hosts Configuration page is displayed.
  3. Search for Enable Cgroups.
  4. Check the Enable Cgroup-based Resource Management box.

cgroups is a Linux kernel feature. cgroups is supported on the following Linux operating systems:

  • CentOS

  • RHEL

  • SUSE

  • Ubuntu

The setup procedure differs depending on the version of cgroups your operating system uses:

  • cgroups v1 — Default on RHEL 7, RHEL 8, and older Ubuntu releases. You must manually create the cgroup directory hierarchy and set ownership before starting the NodeManager.
  • cgroups v2 — Default on RHEL 9 and Ubuntu 22.04 and later. The operating system manages the cgroup hierarchy. Do not create cgroup directories manually or run chown on cgroup paths.

Create the cgroup directory hierarchy (cgroups v1 only)

Run the following commands on every reboot of the NodeManager hosts to set up the cgroup hierarchy. Operating systems use different mount points for the cgroup interface. Replace /sys/fs/cgroup with your operating system equivalent.

mkdir -p /sys/fs/cgroup/cpu/yarn
chown -R yarn /sys/fs/cgroup/cpu/yarn
mkdir -p /sys/fs/cgroup/memory/yarn
chown -R yarn /sys/fs/cgroup/memory/yarn
mkdir -p /sys/fs/cgroup/blkio/yarn
chown -R yarn /sys/fs/cgroup/blkio/yarn
mkdir -p /sys/fs/cgroup/net_cls/yarn
chown -R yarn /sys/fs/cgroup/net_cls/yarn
mkdir -p /sys/fs/cgroup/devices/yarn
chown -R yarn /sys/fs/cgroup/devices/yarn
  1. In Cloudera Manager, select the YARN service.
  2. Click the Configuration tab.
  3. Search for Always Use Linux Container Executor and select YARN-1 (Service-Wide) option.
  4. Search for NodeManager Advanced Configuration and set the following properties in the NodeManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml field.

    The following properties apply to both cgroups v1 and v2:

    Name: yarn.nodemanager.linux-container-executor.group
    Value: hadoop
    Name: yarn.nodemanager.linux-container-executor.cgroups.mount
    Value: false

    The following properties are required for cgroups v2 only:

    Name: yarn.nodemanager.linux-container-executor.cgroups.v2.enabled
    Value: true
    Name: yarn.nodemanager.linux-container-executor.cgroups.mount-path
    Value: /sys/fs/cgroup
  5. Search for CGroups and select YARN-1 (Service-Wide) option in the Use CGroups for Resource Management field.
  6. Search for CGroups Hierarchy and set the NodeManager Default Group value to /hadoop-yarn.
  7. (Optional) Set the percentage of CPU to be used by YARN. Search for Containers CPU Limit and set the value in the Containers CPU Limit Percentage field.
    Set the percentage of CPU that can be allocated for YARN containers. In most cases, the default value of 100% should be used. If you have another process that needs to run on a node that also requires CPU resources, you can lower the percentage of CPU allocated to YARN to free up resources for the other process.
  8. (Optional) Set flexible or strict CPU limits. Search for Strict CGroup Resource Usage and select the NodeManager Default Group field.

    CPU jobs are constrained with CPU scheduling and cgroups enabled, but by default these are flexible limits. If spare CPU cycles are available, containers are allowed to exceed the CPU limits set for them. With flexible limits, the amount of CPU resources available for containers to use can vary based on cluster usage -- the amount of CPU available in the cluster at any given time.

    You can use cgroups to set strict limits on CPU usage. When strict limits are enabled, each process receives only the amount of CPU resources it requests. With strict limits, a CPU process will receive the same amount of cluster resources every time it runs.

    Strict limits are not enabled (set to false) by default.