Troubleshoot cgroups configuration

Use this reference to resolve common cgroups configuration issues, including required default property values and legacy cgroups v1 subhierarchy cleanup after an OS upgrade.

Required default configurations

The following properties must remain at their default values for cgroups to function correctly. Changing these values causes the NodeManager to fail to start or to apply incorrect resource isolation.

yarn.nodemanager.linux-container-executor.group
Must be set to hadoop. This property defines the Unix group that owns the container-executor binary and configuration files. Changing this value breaks the setuid security model and prevents containers from starting.
yarn.nodemanager.linux-container-executor.cgroups.mount
Must be set to false. Automatic mounting of the cgroup hierarchy by the NodeManager is not supported. The cgroup filesystem must be mounted by the operating system before the NodeManager starts.

OS upgrade cleanup after migration to cgroups v2

If you performed an in-place major OS upgrade to a cgroups v2-only environment (for example, from RHEL 7 or RHEL 8 to RHEL 9), legacy cgroups v1 subhierarchies might remain on disk. Due to YARN-11813, YARN detects these legacy paths and incorrectly falls back to cgroups v1, breaking resource isolation on the upgraded OS.

Detecting legacy paths

Check for the existence of the following legacy cgroups v1 directories on each NodeManager host:

  • /sys/fs/cgroup/cpu
  • /sys/fs/cgroup/memory
  • /sys/fs/cgroup/blkio
  • /sys/fs/cgroup/net_cls
  • /sys/fs/cgroup/devices

If any of these paths exist, complete the following cleanup procedure before starting the NodeManager.

Remediating legacy paths

  1. For each legacy directory that contains a cgroup.procs file, migrate all process IDs from that file to the global cgroups v2 procs file. For example:
    cat /sys/fs/cgroup/cpu/cgroup.procs >> /sys/fs/cgroup/cgroup.procs
    Repeat for each legacy path that contains a cgroup.procs file.
  2. Recursively remove all nested subdirectories (such as yarn or containers) from the bottom up within each legacy directory.
  3. Delete the top-level legacy cgroup directories listed above.