Manage Cluster Capacity with Queues

You can manage your cluster capacity using queues to balance resource requirements of multiple applications from various users.

You can use the Capacity Scheduler to share cluster resources using FIFO (first-in, first-out) queues. YARN allows you to configure queues to own a fraction of the capacity of each cluster, and this specified queue capacity is fulfilled dynamically from the available nodes.

Users can submit applications to different queues at multiple levels in the queue hierarchy if the capacity is available on the nodes in the cluster. Because total cluster capacity can vary, capacity configuration values are expressed using percentages.

Specify the capacity property to allocate a floating-point percentage values of cluster capacity to a queue. The following properties divide the cluster resources between the Engineering, Support, and Marketing organizations in a 6:1:3 ratio (60%, 10%, and 30%).

Property: yarn.scheduler.capacity.root.engineering.capacity

Value: 60

Property: yarn.scheduler.capacity.root.support.capacity

Value: 10

Property: yarn.scheduler.capacity.root.marketing.capacity

Value: 30

If you want the Engineering group to split its capacity between the Development and QA sub-teams in a 1:4 ratio. You can set the following property values:

Property: yarn.scheduler.capacity.root.engineering.development.capacity

Value: 20

Property: yarn.scheduler.capacity.root.engineering.qa.capacity

Value: 80

If you want the Engineering, Support, and Marketing organizations to use a specified absolute value for each resource type, you can set the following property values where the Engineering, Support, and Marketing queues are each allocated 10 GB of memory and 12 vcores, and 4 GPU cores:

Property: yarn.scheduler.capacity.root.engineering.capacity

Value: [memory=10240,vcores=12,yarn.io/gpu=4]

Property: yarn.scheduler.capacity.root.support.capacity

Value: [memory=10240,vcores=12,yarn.io/gpu=4]

Property: yarn.scheduler.capacity.root.marketing.capacity

Value: [memory=10240,vcores=12,yarn.io/gpu=4]

Note
The resource value of the parent queue is used if you do not provide a memory or a vcore value.

	Note
The resource value of the parent queue is used if you do not provide a memory or a vcore value.

If you want to enable resource elasticity

Specify the maximum capacity as a floating-point percentage value of resources allocated for a queue. You have to set the maximum capacity to be higher than or equal to the absolute capacity for each queue. Setting this value to -1 sets maximum capacity to 100%. In the following example, the maximum capacity of the Engineering queue is set as 70%.

Property: yarn.scheduler.capacity.root.engineering.maximum-capacity

Value: 70