Set up Queues

Capacity Scheduler queues can be set up in a hierarchy that reflects the database structure, resource requirements, and access restrictions required by the various organizations, groups, and users that utilize cluster resources.

The fundamental unit of scheduling in YARN is a queue. The capacity of each queue specifies the percentage of cluster resources that are available for applications submitted to the queue.

For example, suppose that a company has three organizations: Engineering, Support, and Marketing. The Engineering organization has two sub-teams: Development and QA. The Support organization has two sub-teams: Training and Services. And finally, the Marketing organization is divided into Sales and Advertising. The following image shows the queue hierarchy for this example:

Each child queue is tied to its parent queue with the yarn.scheduler.capacity.<queue-path>.queues configuration property in the capacity-scheduler.xml file. The top-level "support", "engineering", and "marketing" queues would be tied to the "root" queue as follows:

Property: yarn.scheduler.capacity.root.queues

Value: support,engineering,marketing

Example:

<property>
 <name>yarn.scheduler.capacity.root.queues</name>
 <value>support,engineering,marketing</value>
 <description>The top-level queues below root.</description>
</property>

Similarly, the children of the "support" queue would be defined as follows:

Property: yarn.scheduler.capacity.support.queues

Value: training,services

Example:

<property>
 <name>yarn.scheduler.capacity.support.queues</name>
 <value>training,services</value>
 <description>child queues under support</description>
</property>

The children of the "engineering" queue would be defined as follows:

Property: yarn.scheduler.capacity.engineering.queues

Value: development,qa

Example:

<property>
 <name>yarn.scheduler.capacity.engineering.queues</name>
 <value>development,qa</value>
 <description>child queues under engineering</description>
</property>

And the children of the "marketing" queue would be defined as follows:

Property: yarn.scheduler.capacity.marketing.queues

Value: sales,advertising

Example:

<property>
 <name>yarn.scheduler.capacity.marketing.queues</name>
 <value>sales,advertising</value>
 <description>child queues under marketing</description>
</property>