Cloudera Docs

Fair Scheduler features and conversion details

Certain Fair Scheduler properties cannot be auto-converted by the fs2cs conversion utility. Review the list of these properties and if they are supported in Capacity Scheduler and by Queue Manager UI to learn how you can configure them.

Table 1. Queue resource-quota related features
Property Description Conversion information
<minResources> Minimum resources the queue is entitled to.

Partially supported in Capacity Scheduler.

Ignored by the fs2cs conversion utility.

Not supported by Queue Manager UI.
<maxResources> Maximum amount of resources that will be allocated to a queue.

There is an equivalent feature in Capacity Scheduler.

Ignored by the fs2cs conversion utility. For each queue, max-capacity will be set to 100%.

Supported by Queue Manager UI.
<maxChildResources> Maximum amount of resources that can be allocated to an ad hoc child queue.

There is an equivalent feature in Capacity Scheduler.

Ignored by the fs2cs conversion utility. Its value can be two distinct percentages (vcore/memory) or an absolute resources, but the leaf-queue-template only accepts a single percentage.

Supported by Queue Manager UI.
<schedulingPolicy> Scheduling policy of a queue (for example, how submitted applications should be ordered over time). .

There is an equivalent feature in Capacity Scheduler.

Manual fine tuning might be necessary.

Supported by Queue Manager UI.
Table 2. Queue resource-quota related features
Property Description Conversion information
<minResources> Minimum resources the queue is entitled to.

Partially supported in Capacity Scheduler.

Ignored by the fs2cs conversion utility.

Not supported by Queue Manager UI.
<maxResources> Maximum amount of resources that will be allocated to a queue.

There is an equivalent feature in Capacity Scheduler.

Ignored by the fs2cs conversion utility. For each queue, max-capacity will be set to 100%.

Supported by Queue Manager UI.
<maxChildResources> Maximum amount of resources that can be allocated to an ad hoc child queue.

There is an equivalent feature in Capacity Scheduler.

Ignored by the fs2cs conversion utility. Its value can be two distinct percentages (vcore/memory) or an absolute resources, but the leaf-queue-template only accepts a single percentage.

Supported by Queue Manager UI.
Table 3. Global scheduling settings
Property Description Conversion information
<user name="..."> <maxRunningApps>...</maxRunningApps></user> Maximum running apps per user

There is an equivalent feature in Capacity Scheduler.

Fine-tuning of the following three properties are required:
  • Maximum apps per queue
  • User limit percent
  • User limit factor

Supported by Queue Manager UI.
<userMaxAppsDefault> Default maximum running apps

Not supported in Capacity Scheduler.
yarn.scheduler.fair.max.assign Dynamic maximum assign

There is an equivalent feature in Capacity Scheduler.

Fine-tuning of the following three properties are required:
  • yarn.scheduler.capacity.per-node-heartbeat.multiple-assignments-enable
  • yarn.scheduler.capacity.per-node-heartbeat.maximum-container-assignments
  • yarn.scheduler.capacity.per-node-heartbeat.maximum-offswitch-assignments

Supported by Queue Manager UI.
yarn.scheduler.fair.user-as-default-queue User as default queue
There is a very similar feature in Capacity Scheduler. Perform the following steps:
  1. Create a queue, such as root.users and enable the suto-create-child-queue setting for it.
  2. Use the following placement rule: "u%user:%user"
The following restrictions apply:
  • It is not possible to have root as a parent for dynamically created queues.
  • root.users cannot have static leafs, that is, queues that are defined in the capacity-scheduler.xml file.

For more information, see the Placement Rules table.

Supported by Queue Manager UI.
Table 4. Global scheduling settings
Property Description Conversion information
yarn.scheduler.fair.max.assign Dynamic maximum assign

There is an equivalent feature in Capacity Scheduler.

Fine-tuning of the following three properties are required:
  • yarn.scheduler.capacity.per-node-heartbeat.multiple-assignments-enable
  • yarn.scheduler.capacity.per-node-heartbeat.maximum-container-assignments
  • yarn.scheduler.capacity.per-node-heartbeat.maximum-offswitch-assignments

Not supported by Queue Manager UI.
yarn.scheduler.fair.user-as-default-queue User as default queue
Relative mode:A placement rule needs to be created.
  1. Create a queue, such as "root.users" and enable Dynamic Auto Child Creation for it (make it a Managed Parent Queue).
  2. Create the following placement rule:
    {
  "type":"user",
  "matches":"*",
  “parentQueue”: “root.users”,
  "policy":"user",
  “create”: true,
  "fallbackResult":"skip"
}
The following limitations apply:
  • It is not possible to have "root" as a parent for dynamically created queues.
  • "root.users" queue cannot have static leafs. Those are queues that always exist and are created manually.

For information about weight mode see Auto-converted Fair Scheduler properties.

Supported by Queue Manager UI.
Table 5. Preemption
Property Description Conversion information
yarn.scheduler.fair.preemption.cluster-utilization-threshold The utilization threshold after which preemption kicks in.

There is an equivalent feature in Capacity Scheduler: yarn.resourcemanager.monitor.capacity.preemption.max_ignored_over_capacity. It specifies the resource usage threshold over its configured capacity that a queue must meet before it is eligible for preemption.

Supported by Queue Manager UI.
minSharePreemptionTimeout The number of seconds the queue is under its minimum share before it will try to preempt containers to take resources from other queue.s

Not supported in Capacity Scheduler.
fairSharePreemptionTimeout The number of seconds the queue is under its fair share threshold before it will try to preempt containers to take resources from other queues.

Partially supported in Capacity Scheduler.

This can be achieved by using the following configurations together:
  • yarn.resourcemanager.monitor.capacity.preemption.natural_termination_factor
  • yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill

Supported by Queue Manager UI.
fairSharePreemptionThreshold The fair share preemption threshold for the queue.

Partially supported in Capacity Scheduler.

This can be achieved by using the following configurations together:
  • yarn.resourcemanager.monitor.capacity.preemption.natural_termination_factor
  • yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill

Supported by Queue Manager UI.
Table 6. Placement rules
Fair Scheduler placement rules Description Conversion information
create="false" or "true" Disable or enable creating a queue dynamically in YARN. This option can be specified on all rules.

Partially supported in Capacity Scheduler.

Use the Capacity Scheduler Dynamic Queue Mappings policies:
  • u:%user:[managedParentQueueName].[queueName]
  • u:%user:[managedParentQueueName].%user
  • u:%user:[managedParentQueueName].%primary_group
  • u:%user:[managedParentQueueName].%secondary_group

Supported by Queue Manager UI.
<rule name="specified"/> If a user has submitted the application by specifying a queue name (other than the “default” queue), then this rule will be successful. Hence the remaining set of rules won't be executed.

Not supported in Capacity Scheduler.
<rule name="primaryGroupExistingQueue"/> If submitted user’s(userA) primary group name (groupA) exists, submit to groupA.

There is an equivalent placement rule in Capacity Scheduler: <value>u:%user:%primary_group</value>

Supported by Queue Manager UI.
<rule name="secondaryGroupExistingQueue"/> If submitted user’s(userA) secondary group name (groupA) exists, submit to groupA.

There is an equivalent placement rule in Capacity Scheduler: <value>u:%user:%secondary_group</value>

Supported by Queue Manager UI.
<rule name="nestedUserQueue">
Depending on the nested rule, this places the job to the following queues:
  • root.[primaryGroup].[userName]
  • root.[secondaryGroup].[userName]
  • root.[queuePath].[userName]

Not supported in Capacity Scheduler.
<rule name="default" queue=”qName”/> Fall back policy by which rule will fall back to queue named in the property ‘queue’ or the “default“ queue if no queue property is specified (if all matches fail).

There is an equivalent placement rule in Capacity Scheduler: <value>u:%user:default</value>

Supported by Queue Manager UI.