Set queue priorities

For long-running applications and applications that required large containers, you must enable preemption for the YARN queue priorities to be properly applied.

Even with preemption enabled, there are some use cases where applications might not have access to cluster resources without setting priorities:

Long-running applications – Without setting priorities, long-running applications in queues that are under capacity and with lower relative resource usage may not release cluster resources until they finish running.
Applications that require large containers – The issue with long-running applications is exacerbated for applications that require large containers. With short-running applications, previous containers may eventually finish running and free cluster resources for applications with large containers. But with long-running services in the cluster, the large containers may never get sufficiently large resources on any nodes.
Hive LLAP – Hive LLAP (Low-Latency Analytical Processing) enables you to run Hive queries with low-latency in near real-time. To ensure low-latency, you should set the priority of the queue used for LLAP to a higher priority, especially if your cluster includes long-running applications.

note

To set the queue used for Hive LLAP, select Hive > Config > Settings on the Ambari dashboard, then select a queue using the Interactive Query Queue drop-down. For more information, see the Hive Performance Tuning guide.

For example, the following figure shows a 3-node cluster with long-running 20 GB containers. The LLAP daemons require 90 GB of cluster resources, but preemption does not occur because the available queues are under capacity with lower relative resource usage. With only 80 GB available on any of the nodes, LLAP must wait for the long-running applications to finish before it can access cluster resources.

Prerequisites

To set the queue priority, perform the following:

In Cloudera Manager, select the YARN service.
Click the Configuration tab.
Search for scheduler.
In Capacity Scheduler Configuration Advanced Configuration Snippet (Safety Valve)add the following:
```
Name: yarn.scheduler.capacity.<queue-path>.priority
Value: 2
```
All queues are set to a priority of 0 by default. Higher numbers indicate higher priority.