Preemption allows applications of higher priority to preempt applications of lower
priority.
A scenario can occur where a queue has a guaranteed level of cluster resources, but must
wait to run applications because other queues are utilizing all of the available resources.
If preemption is enabled, applications of higher priority do not have to wait because
applications of lower priority have taken up the available capacity. When you enable
Preemption (yarn.resourcemanager.scheduler.monitor.enable) underserved queues can
begin to claim their allocated cluster resources almost immediately, without having to wait
for other queues' applications to finish running.
You can use Priority Scheduling to run YARN applications at a higher priority, regardless
of other applications that are already running in the cluster . For more information, see
Setting global maximum application priority.
To configure the Queue Preemption options on all queues, perform the following:
-
In Cloudera Manager, navigate to .
-
Search for preemption.
-
Find the Capacity Scheduler Preemption property.
-
Ensure that it is checked, meaning that Preemption is enabled.
-
Under Configuration, search for
ResourceManager Advanced
Configuration Snippet (Safety Valve) for yarn-site.xml.
-
Configure the required preemption properties:
- Preemption: Observe Only - Add the
yarn.resourcemanager.monitor.capacity.preemption.observe_only
parameter and the value you need.
It runs the policy, but does not affect the
cluster with preemption and stop events.
- Preemption: Monitoring Interval (ms) - Add the
yarn.resourcemanager.monitor.capacity.preemption.monitoring_interval
parameter and the value you need.
The time in milliseconds between invocations of
this policy. Setting this value to a longer time interval causes the Capacity
Monitor to run less frequently.
- Preemption: Maximum Wait Before Kill (ms) - Add the
yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill
parameter and the value you need.
The time in milliseconds between requesting a
preemption from an application and stopping the container. Setting this to a higher
value gives applications more time to respond to preemption requests and gracefully
release containers.
- Preemption: Total Resources Per Round - Add the
yarn.resourcemanager.monitor.capacity.preemption.total_preemption_per_round
parameter and the value you need.
The maximum percentage of resources preempted in a
single round. You can use this value to restrict the pace at which containers are
reclaimed from the cluster. After computing the total desired preemption, the policy
scales it back to this limit.
- Preemption: Over Capacity Tolerance - Add the
yarn.resourcemanager.monitor.capacity.preemption.max_ignored_over_capacity
parameter and the value you need.
The maximum amount of resources above the target
capacity ignored for preemption. The default value is 0.1, which means that the
Resource Manager starts preemption for a queue only when it goes 10% above its
guaranteed capacity. This avoids resource-rotation and aggressive
preemption.
- Preemption: Maximum Termination Factor - Add the
yarn.resourcemanager.monitor.capacity.preemption.natural_termination_factor
parameter and the value you need.
The maximum percentage of each queue's preemption
target capacity that is preempted per cycle. You can increase this value to speed up
resource reclamation.
-
Click Save Changes.
-
Restart the YARN services.
