Allocating Resources with Capacity Scheduler

The Capacity Scheduler enables multiple users and groups to share allocated cluster resources in a predictable and timely manner.

Capacity Scheduler Overview
You can use the Capacity Scheduler to allocate shared cluster resources among users and groups.
Enable the Capacity Scheduler
To enable the Capacity Scheduler using Cloudera Manager:
Set up queues
Capacity Scheduler queues can be set up in a hierarchy that reflects the database structure, resource requirements, and access restrictions required by the various organizations, groups, and users that utilize cluster resources.
Control access to queues with ACLs
Use Access-control lists (ACLs) to control user and administrator to Capacity Scheduler queues.
Define queue mapping policies
Administrators can define a default mapping policy to specify that applications submitted by users are automatically submitted to queues.
Manage cluster capacity with queues
You can manage your cluster capacity using queues to balance resource requirements of multiple applications from various users.
Set queue priorities
For long-running applications and applications that required large containers, you must enable preemption for the YARN queue priorities to be properly applied.
Resource distribution workflow
During scheduling, queues at any level in the hierarchy are sorted in the order of their current used capacity, and available resources are distributed among them starting with queues that are currently the most under-served.
Resource distribution workflow example
To understand the resource distribution workflow, consider the example of a 100-node cluster, each with 10 GB of memory allocated for YARN containers, for a total cluster capacity of 1000 GB (1 TB).
Set user limits
Set a minimum percentage of resources allocated to each leaf queue user.
Application reservations
For a resource-intensive application, the Capacity Scheduler creates a reservation on a cluster node if the node's free capacity can meet the particular application's requirements. This ensures that the resources are utilized only by that particular application until the application reservation is fulfilled.
Set flexible scheduling policies
Set FIFO (First-In, First-Out) or Fair scheduling policies in Capacity Scheduler depending on your requirements.
Start and stop queues
Queues in YARN can be in two states: RUNNING or STOPPED. A RUNNING state indicates that a queue can accept application submissions, and a STOPPED queue does not accept application submissions. The default state of any configured queue is RUNNING.
Set application limits
To avoid system-thrash due to an unmanageable load -- caused either by malicious users, or by accident -- the Capacity Scheduler enables you to place a static, configurable limit on the total number of concurrently active (both running and pending) applications at any one time.
Enable preemption
Capacity Scheduler Preemption allows higher-priority applications to preempt lower-priority applications.
Enable priority scheduling
You can use Priority Scheduling to run YARN applications at higher priority, regardless of other applications that are already running in the cluster. YARN allocates more resources to applications running at a higher priority over those running at a lower priority. Priority Scheduling enables you to set an application's priority both at the time of submission and dynamically at run time.
Configure ACLs for application priorities
Configure Priority ACLs to ensure that only select users can submit applications with a specified priority to a queue. You must configure these Priority ACLs at the leaf queue-level.
Enable intra-queue preemption
Intra-queue preemption helps in effective distribution of resources within a queue based on configured user limits or application priorities.