Configuring preemption

Preemption allows applications of higher priority to preempt applications of lower priority.

A scenario can occur where a queue has a guaranteed level of cluster resources, but must wait to run applications because other queues are utilizing all of the available resources. If preemption is enabled for a particular queue, applications of higher priority do not have to wait because applications of lower priority have taken up the available capacity. When you enable Preemption (yarn.resourcemanager.scheduler.monitor.enable) the allocated cluster resources are taken from the preemption enabled queue utilizing more than their guaranteed capacity and served to the underserved queues immediately, without having to wait for other queues' applications to finish running.

You can use Priority Scheduling to run YARN applications at a higher priority, regardless of other applications that are already running in the cluster . For more information, see Setting global maximum application priority.

To configure the Queue Preemption options on all queues, perform the following:

In Cloudera Manager, navigate to YARN > Configuration.
Search for preemption.
Find the Capacity Scheduler Preemption property.
Ensure that it is checked, meaning that Preemption is enabled.
Under Configuration, search for ResourceManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml.
Configure the required preemption properties:
- Preemption: Observe Only - Add the yarn.resourcemanager.monitor.capacity.preemption.observe_only parameter and the value you need.
  It runs the policy, but does not affect the cluster with preemption and stop events.
- Preemption: Monitoring Interval (ms) - Add the yarn.resourcemanager.monitor.capacity.preemption.monitoring_interval parameter and the value you need.
  The time in milliseconds between invocations of this policy. Setting this value to a longer time interval causes the Capacity Monitor to run less frequently.
- Preemption: Maximum Wait Before Kill (ms) - Add the yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill parameter and the value you need.
  The time in milliseconds between requesting a preemption from an application and stopping the container. Setting this to a higher value gives applications more time to respond to preemption requests and gracefully release containers.
- Preemption: Total Resources Per Round - Add the yarn.resourcemanager.monitor.capacity.preemption.total_preemption_per_round parameter and the value you need.
  The maximum percentage of resources preempted in a single round. You can use this value to restrict the pace at which containers are reclaimed from the cluster. After computing the total desired preemption, the policy scales it back to this limit.
- Preemption: Over Capacity Tolerance - Add the yarn.resourcemanager.monitor.capacity.preemption.max_ignored_over_capacity parameter and the value you need.
  The maximum amount of resources above the target capacity ignored for preemption. The default value is 0.1, which means that the Resource Manager starts preemption for a queue only when it goes 10% above its guaranteed capacity. This avoids resource-rotation and aggressive preemption.
- Preemption: Maximum Termination Factor - Add the yarn.resourcemanager.monitor.capacity.preemption.natural_termination_factor parameter and the value you need.
  The maximum percentage of each queue's preemption target capacity that is preempted per cycle. You can increase this value to speed up resource reclamation.
Click Save Changes.
Restart the YARN services.
For information about configuring preemption for a specific queue, see Configure preemption for a specific queue.