Cloudera Docs

Configure preemption

Preemption allows applications of higher priority to preempt applications of lower priority.

A scenario can occur in which a queue has a guaranteed level of cluster resources, but must wait to run applications because other queues are utilizing all of the available resources. If preemption is enabled, applications of higher priority do not have to wait because applications of lower priority have taken up the available capacity. When you enable Preemption (yarn.resourcemanager.scheduler.monitor.enable) underserved queues can begin to claim their allocated cluster resources almost immediately, without having to wait for other queues' applications to finish running.

You can use Priority Scheduling to run YARN applications at a higher priority, regardless of other applications that are already running in the cluster . For more information, see Set global maximum application priority.

To configure the Queue Preemption options on all queues, perform the following:

  1. In Cloudera Manager, select the Clusters > YARN Queue Manager UI service. A graphical queue hierarchy is displayed in the Overview tab.
  2. Click the Scheduler Configuration tab.
  3. Select the Enable Monitoring Policies checkbox.
  4. Specify org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy as the monitoring policy to be used in the Monitoring Policies textbox.
  5. Configure the required preemption properties:
    • Preemption: Observe Only - Select the checkbox to run the policy, but not affect the cluster with preemption and kill events.
    • Preemption: Monitoring Interval (ms) - The time in milliseconds between invocations of this policy. Setting this value to a longer time interval causes the Capacity Monitor to run less frequently.
    • Preemption: Maximum Wait Before Kill (ms) - The time in milliseconds between requesting a preemption from an application and killing the container. Setting this to a higher value gives applications more time to respond to preemption requests and gracefully release containers.
    • Preemption: Total Resources Per Round - The maximum percentage of resources preempted in a single round. You can use this value to restrict the pace at which containers are reclaimed from the cluster. After computing the total desired preemption, the policy scales it back to this limit.
    • Preemption: Over Capacity Tolerance - The maximum amount of resources above the target capacity ignored for preemption. The default value is 0.1, which means that the Resource Manager starts preemption for a queue only when it goes 10% above its guaranteed capacity. This avoids resource-rotation and aggressive preemption.
    • Preemption: Maximum Termination Factor - The maximum percentage of each queue's preemption target capacity that is preempted per cycle. You can increase this value to speed up resource reclamation.
  6. Click Save.

    For information about configuring preemption for a specific queue, see Configure preemption for a specific queue.