Cloudera Docs

Auto-converted Fair Scheduler properties

The fs2cs conversion utility automatically converts certain Fair Scheduler properties into Capacity Scheduler properties. Reviewing the list of auto-converted properties enables you to verify the conversion and plan the manual fine-tuning that requires to be done after the upgrade is completed.

Loading Filters ... 7.7.1 7.6.7 7.6.1 7.5.1 7.4.4 7.3.1 7.2.4 7.1.4 7.1.3 7.1.2 7.1.1 7.0.3 5.16 5.15 5.14 5.13 5.16 5.15 5.14 5.13 7.1.8 7.1.7.1000 7.1.7 7.1.6 7.1.5 7.1.4 7.1.3 7.1.2 7.1.1 7.1.7.2000 7.7.1 7.6.7 7.6.1 7.5.1 7.4.4 7.4.3 7.3.1 7.2.4 7.1.4 7.1.3 7.1.2 7.1.1

Table 1. Queue resource-quota related features
Property Description

Pre-created hierarchical queues.

The same queue hierarchy is achieved after conversion.
<weight>

Weight: The steady fair share of a queue.

The queue.capacity property will be set with the same ratio.
<maxAMShare>

Maximum AM share: Limits the fraction of the queue’s fair share that can be used to run application masters

<maxRunningApps>

Maximum running apps: Limits the number of apps from the queue to run at once
<maxContainerAllocation>

Maximum container allocation: Maximum amount of resources a queue can allocate for a single container.
<schedulingPolicy>

Scheduling policy of a queue (for example, how submitted applications are ordered over time).

It is converted with some limitations. For more information, see Fair Scheduler features and the conversion details.

<aclSubmitApps> <aclAdministerApps>

ACL settings: List of users and/or groups that can submit apps to the queue or can administer a queue.
maximum-am-resource-percent

Specifies the maximum percentage of resources in the cluster which can be used to run application masters for the queue.
acl_submit_applications

Specifies the ACL which controls who can submit applications to the given queue.
acl_administer_queue

Specifies the ACL which controls who can administer applications in the given queue.
ordering-policy

Specifies the queue ordering policies to FIFO or fair on the given queue.
Table 2. Global scheduling settings
Property Description
yarn.scheduler.fair.allow-undeclared-pools

Allow undeclared pools.

Sets whether new queues can be created at application submission time.
yarn.scheduler.fair.sizebasedweight

Size based weight.

Whether to assign shares to individual apps based on their size, rather than providing an equal share to all apps regardless of size.
<queueMaxAppsDefault>

Queue max apps default: Sets the default running app limit for all queues.
<queueMaxAMShareDefault>

Default max AM share: Sets the default AM resource limit for queue.
yarn.scheduler.fair.locality.threshold.node

Locality threshold node: For applications that request containers on particular nodes, the number of scheduling opportunities since the last container assignment to wait before accepting a placement on another node.
yarn.scheduler.fair.locality.threshold.rack

Locality threshold rack: For applications that request containers on particular racks, the number of scheduling opportunities since the last container assignment to wait before accepting a placement on another rack.
yarn.scheduler.fair.max.assign

Maximum assignments: If assignmultiple is true and dynamic.max.assign is false, the maximum amount of containers that can be assigned in one heartbeat.
yarn.scheduler.fair.assignmultiple

Assign multiple: Whether to allow multiple container assignments in one heartbeat.
yarn.resourcemanager.scheduler.monitor.enable

Allows higher-priority applications to preempt lower-priority applications.
yarn.scheduler.capacity.maximum-am-resource-percent

Specifies the maximum percentage of resources in the cluster which can be used to run application masters.
Table 3. Global scheduling settings
Property Description
yarn.scheduler.fair.allow-undeclared-pools

Allow undeclared pools.

Sets whether new queues can be created at application submission time.
yarn.scheduler.fair.sizebasedweight

Size based weight.

Whether to assign shares to individual apps based on their size, rather than providing an equal share to all apps regardless of size.
<queueMaxAppsDefault>

Queue max apps default: Sets the default running app limit for all queues.
<queueMaxAMShareDefault>

Default max AM share: Sets the default AM resource limit for queue.
yarn.scheduler.fair.locality.threshold.node

Locality threshold node: For applications that request containers on particular nodes, the number of scheduling opportunities since the last container assignment to wait before accepting a placement on another node.
yarn.scheduler.fair.locality.threshold.rack

Locality threshold rack: For applications that request containers on particular racks, the number of scheduling opportunities since the last container assignment to wait before accepting a placement on another rack.
yarn.scheduler.fair.max.assign

Maximum assignments: If assignmultiple is true and dynamic.max.assign is false, the maximum amount of containers that can be assigned in one heartbeat.
yarn.scheduler.fair.assignmultiple

Assign multiple: Whether to allow multiple container assignments in one heartbeat.
yarn.resourcemanager.scheduler.monitor.enable

Allows higher-priority applications to preempt lower-priority applications.
yarn.scheduler.capacity.maximum-am-resource-percent

Specifies the maximum percentage of resources in the cluster which can be used to run application masters.
<userMaxAppsDefault>

Default maximum running applications.
<user name="..."> <maxRunningApps>...</maxRunningApps></user>

Maximum running applications per user.
yarn.scheduler.fair.user-as-default-queue

Whether to use the username associated with the allocation as the default queue name.

Weight mode: This behavior is simulated with a placement rule (in fact, even in Fair Scheduler, this is translated into a placement rule internally):
{
  "type":"user",
  "matches":"*",
  “parentQueue”: “root”,
  "policy":"user",
  “create”: true,
  "fallbackResult":"skip"
}

For information about percentage mode, see Fair Scheduler features ad conversion details.
Table 4. Preemption
Property Description
yarn.scheduler.fair.preemption

Fair Scheduler preemption turned on.

After the conversion capacity Scheduler preemption is turned on by default using the default values.
<allowPreemptionFrom>

Per-queue preemption disabled.

After the conversion the same queue preemption disabled by default.
yarn.scheduler.fair.waitTimeBeforeKill

Wait time before killing a container
disable_preemption

Disables preemption of application containers submitted to a given queue.
Table 5. Placement rules
Fair Scheduler placement rules Description Conversion details
create="false" or "true"
Disable or enable creating a queue dynamically in YARN. This option cannot be specified on the following placement rule policies:
  • reject
  • setDefaultQueue
  • defaultQueue

Weight mode: This flag is fully supported, except for nested rules, where you can define a single “create” flag only. Therefore, “true/false” and “false/true” cannot be set.

Relative mode: Partially supported. A managed parent queue must be chosen as a parent. The flag has no effect on regular parent queues.
<rule name="specified"/> If a user has submitted the application by specifying a queue name (other than the “default” queue), then this rule will be successful. Hence the remaining set of rules won't be executed.

Supported in both weight and percentage mode.
<rule name="primaryGroup"/>

If the submitted user’s(userA) primary group name (groupA) exists, submit to groupA.

The matching policy is called primaryGroup.
<rule name="secondaryGroupExistingQueue"/>

If the submitted user’s(userA) secondary group name (groupB) exists, submit to groupB.

The matching policy is called secondaryGroup.
<rule name="nestedUserQueue">
Depending on the nested rule, this places the job to the following queues:
  • root.[primaryGroup].[userName]
  • root.[secondaryGroup].[userName]
  • root.[queuePath].[userName]

Supported by Capacity Scheduler.

The three possible policies are (depending on the outer rule):

  • primaryGroupUser
  • secondaryGroupUser
  • user with a parentQueue set explicitly.
<rule name="default" queue=”qName”/>

Places the application into the default queue called “root.default” or to a user-specific one denoted by the “queue” attribute.

The default rule has a matching policy called defaultQueue.

If “root.default” is not the intended default queue, then two approaches are possible:
  • Use the setDefaultQueue policy to change “root.default”, then apply defaultQueue.
  • Use the custom policy with the policy string being set to the target queue.