Configuring the Fair Scheduler
The Fair Scheduler is the Cloudera recommended scheduler option. The Fair Scheduler controls how resources are allocated to pools (or queues) and how jobs are assigned to pools. Jobs can also be explicitly submitted to pools; to submit an job to a specific pool, you specify the mapreduce.job.queuename property.
Pools have policies for preempting running jobs, so that when there is contention for resources, jobs that have high priority or have been waiting a long time to run are allowed to run.
Fair Scheduler configuration is maintained in two files: yarn-site.xml and fair-scheduler.xml. Detailed information on the available properties is available at Fair Scheduler Configuration. When you change the contents of yarn-site.xml, you must restart the YARN service.
|yarn.scheduler.fair.allow-undeclared-pools||When set to true, the Fair Scheduler uses the username as the default pool name, in the event that a pool
name is not specified. When set to false, all applications are run in a shared pool, called default.
|yarn.scheduler.fair.user-as-default-queue||When set to true, pools specified in applications but not explicitly configured, are created at runtime
with default settings. When set to false, applications specifying pools not
explicitly configured run in a pool named default. This setting applies when an application explicitly specifies a pool and when the application runs in a pool named
with the username associated with the application.
|yarn.scheduler.fair.preemption||When enabled, under certain conditions, the Fair Scheduler preempts applications in other pools. Preemption guarantees that production
applications are not starved while also allowing the cluster to be used for experimental and research applications. To minimize wasted computation, the Fair Scheduler preempts the most recently
|yarn.scheduler.fair.preemption.cluster-utilization-threshold||The cluster utilization threshold above which preemption is triggered. If the cluster utilization is under this threshold, preemption
is not triggered even if there are starved queues. The utilization is computed as the maximum ratio of usage to capacity among all resources.
... <property> <name>yarn.scheduler.fair.allow-undeclared-pools</name> <value>true</value> </property> <property> <name>yarn.scheduler.fair.user-as-default-queue</name> <value>true</value> </property> <property> <name>yarn.scheduler.fair.preemption</name> <value>true</value> </property> <property> <name>yarn.scheduler.fair.preemption.cluster-utilization-threshold</name> <value>0.8</value> </property> ...
|queuePlacementPolicy||Policy for assigning jobs to resource pools. In Cloudera Manager this property is configured using placement rules.|
|userMaxAppsDefault||Default running app limit for a user whose limit is not otherwise specified. In Cloudera Manager this property is configured using user limits.|
|queueMaxAppsDefault||Default running app limit for pools; overridden by the equivalent element in a pool.|
|queueMaxAMShareDefault||Default ApplicationMaster resource limit for the pool; overridden by the equivalent element in a pool.|
|defaultFairSharePreemptionThreshold||Fair share preemption threshold for pools; overridden by the equivalent element in a pool. The threshold value is between 0 and 1. If set to x and the fair share of the pool is F, resources are preempted from other pools if the allocation is less than (x * F).|
|defaultFairSharePreemptionTimeout||Default number of seconds a resource pool is under its fair share before it will preempt containers to take resources from other resource pools; overridden by the equivalent element in a pool. If this parameter is not set, and fairSharePreemptionTimeout is not set for a given queue or its ancestor queues, pre-emption by this queue will never occur, even if pre-emption is enabled. Default timeout is 2^64 milliseconds.|
|defaultMinSharePreemptionTimeout||Default number of seconds a resource pool is under its minimum share before it will preempt containers to take resources from other resource pools; overridden by the equivalent element in a pool.|
|defaultQueueSchedulingPolicy||Default scheduling policy for pools; overridden by the equivalent element in a pool.
|queue||Name of a dynamic resource pool.|
|weight||Weight given to the resource pool when determining how to allocate resources relative to other resource pools. In Cloudera Manager this property is configured using configuration sets.|
|schedulingPolicy||Policy to determine how resources are allocated to the resource pool: fair, fifo, or drf.|
|aclSubmitApps||Users and groups that can submit jobs to the pool.|
|aclAdministerApps||Users and groups that can administer the pool.|
|minResources, maxResources||Minimum and maximum share of resources that can allocated to the resource pool in the form X mb, Y vcores. Values computed by the weight settings are limited by (or constrained by) the minimum and maximum values.|
|maxAMShare||Fraction of the resource pool's fair share that can be used to run ApplicationMasters. For example, if set to 1.0, then ApplicationMasters in the pool can take up to 100% of both the memory and CPU fair share. The value of -1.0 disables this feature, and the ApplicationMaster share is not checked. The default value is 0.5.|
|maxRunningApps||See default elements.|
|fairSharePreemptionThreshold||See default elements.|
|fairSharePreemptionTimeout||See default elements.|
|minSharePreemptionTimeout||See default elements.|
<allocations> <queue name="root"> <weight>1.0</weight> <schedulingPolicy>drf</schedulingPolicy> <aclSubmitApps> </aclSubmitApps> <aclAdministerApps>*</aclAdministerApps> <queue name="production"> <minResources>1024 mb, 10 vcores</minResources> <maxResources>5120 mb, 20 vcores</maxResources> <weight>4.0</weight> <schedulingPolicy>drf</schedulingPolicy> <aclSubmitApps>*</aclSubmitApps> <aclAdministerApps>*</aclAdministerApps> </queue> <queue name="development"> <weight>1.0</weight> <schedulingPolicy>drf</schedulingPolicy> <aclSubmitApps>*</aclSubmitApps> <aclAdministerApps>*</aclAdministerApps> </queue> </queue> <defaultQueueSchedulingPolicy>drf</defaultQueueSchedulingPolicy> <queuePlacementPolicy> <rule name="specified" create="true"/> <rule name="user" create="true"/> </queuePlacementPolicy> </allocations>
Dynamic resource pools allow you to configure scheduler properties. See Configuring Default YARN Fair Scheduler Properties.