This chapter describes how to allocate shared CPU and memory resources among users and groups in a Hadoop cluster.
As discussed in the Capacity Scheduler guide, the fundamental unit of scheduling in YARN is the queue. The capacity of each queue specifies the percentage of cluster resources that are available for applications submitted to the queue. Queues can be set up in a hierarchy that reflects the database structure, resource requirements, and access restrictions required by the various organizations, groups, and users that utilize cluster resources. When the default resource calculator (DefaultResourceCalculator) is used, resources are allocated based on memory alone.
CPU scheduling is enabled by using the Dominant Resource Calculator (DominantResourceCalculator) rather than the default resource calculator. The Dominant Resource Calculator is based on the Dominant Resource Fairness (DRF) model of resource allocation.
With the Dominant Resource Calculator, queues are still used to allocate cluster resources, but both CPU and memory are taken into consideration. In the DRF model, resource allocation takes into account the dominant resource required by a process. The Dominant Resource Calculator schedules both CPU-heavy and memory-heavy processes on the same node. CPU-heavy processes (such as Storm-on-YARN) receive more CPU and less memory. Memory-heavy processes (such as MapReduce) receive more memory and less CPU. The DRF scheduler is designed to fairly distribute memory and CPU resources among different types of processes in a mixed-workload cluster.
CPU scheduling represents one aspect of YARN resource management capabilities that includes CGroups, node labels, archival storage, and memory as storage. CGroups should be used with CPU scheduling to constrain and manage CPU processes.
CPU scheduling is only recommended for Linux. Currently there is no isolation mechanism (CGroups equivalent) for Windows, so do not enable CPU scheduling on Windows.
