Troubleshoot cgroups configuration
Use this reference to resolve common cgroups configuration issues, including required default property values and legacy cgroups v1 subhierarchy cleanup after an OS upgrade.
Required default configurations
The following properties must remain at their default values for cgroups to function correctly. Changing these values causes the NodeManager to fail to start or to apply incorrect resource isolation.
- yarn.nodemanager.linux-container-executor.group
- Must be set to
hadoop. This property defines the Unix group that owns thecontainer-executorbinary and configuration files. Changing this value breaks the setuid security model and prevents containers from starting. - yarn.nodemanager.linux-container-executor.cgroups.mount
- Must be set to
false. Automatic mounting of the cgroup hierarchy by the NodeManager is not supported. The cgroup filesystem must be mounted by the operating system before the NodeManager starts.
OS upgrade cleanup after migration to cgroups v2
If you performed an in-place major OS upgrade to a cgroups v2-only environment (for example, from RHEL 7 or RHEL 8 to RHEL 9), legacy cgroups v1 subhierarchies might remain on disk. Due to YARN-11813, YARN detects these legacy paths and incorrectly falls back to cgroups v1, breaking resource isolation on the upgraded OS.
Detecting legacy paths
Check for the existence of the following legacy cgroups v1 directories on each NodeManager host:
- /sys/fs/cgroup/cpu
- /sys/fs/cgroup/memory
- /sys/fs/cgroup/blkio
- /sys/fs/cgroup/net_cls
- /sys/fs/cgroup/devices
If any of these paths exist, complete the following cleanup procedure before starting the NodeManager.
Remediating legacy paths
- For each legacy directory that contains a cgroup.procs
file, migrate all process IDs from that file to the global cgroups v2 procs
file. For example:
Repeat for each legacy path that contains a cgroup.procs file.cat /sys/fs/cgroup/cpu/cgroup.procs >> /sys/fs/cgroup/cgroup.procs - Recursively remove all nested subdirectories (such as yarn or containers) from the bottom up within each legacy directory.
- Delete the top-level legacy cgroup directories listed above.
