Fair Scheduler

Use the Fair Scheduler, if you want to allocate resources to weighted pools, with fair sharing within each pool.

A scheduler determines which jobs run, where and when they run, and the resources allocated to the jobs. The Fair Scheduler is the Cloudera recommended scheduler option. The Fair Scheduler controls how resources are allocated to pools (or queues) and how jobs are assigned to pools. Jobs can also be explicitly submitted to pools; to submit a job to a specific pool, you specify the mapreduce.job.queuename property.

Pools have policies for preempting running jobs, so that when there is contention for resources, jobs that have high priority or have been waiting for a long time to run are allowed to run.

Fair Scheduler configuration is maintained in two files: yarn-site.xml and fair-scheduler.xml. Detailed information on the available properties is available at Fair Scheduler Configuration. When you change the contents of yarn-site.xml, you must restart the YARN service.

In Cloudera Manager, the Dynamic Resource Pools Configuration screen provides an enhanced interface for configuring the Fair Scheduler. In addition to allowing you to configure resource allocation properties, you can define schedules for changing the values of the properties. Cloudera Manager automatically updates Fair Scheduler configuration files according to the schedule.