Create and Configure YARN Capacity Scheduler Queues
Capacity Scheduler queues can be used to allocate cluster resources among users and groups. These settings can be accessed from Ambari > YARN > Configs > Scheduler or in conf/capacity-scheduler.xml.
The following configuration example demonstrates how to set up Capacity Scheduler queues. This example separates short- and long-running queries into two separate queues:
hive1--This queue is used for short-duration queries and is assigned 50% of cluster resources.
hive2--This queue is used for longer-duration queries and is assigned 50% of cluster resources.
The following capacity-scheduler.xml settings are used to implement this configuration:
yarn.scheduler.capacity.root.queues=hive1,hive2 yarn.scheduler.capacity.root.hive1.capacity=50 yarn.scheduler.capacity.root.hive2.capacity=50
Configure usage limits for these queues and their users with the following settings:
yarn.scheduler.capacity.root.hive1.maximum-capacity=50 yarn.scheduler.capacity.root.hive2.maximum-capacity=50 yarn.scheduler.capacity.root.hive1.user-limit=1 yarn.scheduler.capacity.root.hive2.user-limit=1
Setting maximum-capacity to 50 restricts queue users to 50% of the queue capacity with a hard limit. If the maximum-capacity is set to more than 50%, the queue can use more than its capacity when there are other idle resources in the cluster. However, any user can use only the configured queue capacity. The default value of "1" for user-limit means that any single user in the queue can at a maximum occupy 1X the queue's configured capacity. These settings prevent users in one queue from monopolizing resources across all queues in a cluster.
This example is a basic introduction to queues. For more detailed information on allocating cluster resources using Capacity Scheduler queues, see the "Capacity Scheduler" section of the YARN Resource Management Guide.
Setup Using the Ambari Capacity Scheduler View
If you are using Ambari 2.1 or later, queues can be set up using the Ambari Capacity Scheduler View as shown in the following image:
In Ambari, navigate to the administration page.
Click Views > CAPACITY-SCHEDULER > <your_view_name>, and then click Go to instance at the top of your view page.
In your view instance page, select the queue you want to use or create a queue. See the Ambari Views Guide.
To create the scenario that is shown in the following screen capture, select the root queue and add hive1 and hive2 at that level.
