The fundamental unit of scheduling in YARN is a queue. The capacity of each queue specifies the percentage of cluster resources that are available for applications submitted to the queue. Queues can be set up in a hierarchy that reflects the database structure, resource requirements, and access restrictions required by the various organizations, groups, and users that utilize cluster resources.
For example, suppose that a company has three organizations: Engineering, Support, and Marketing. The Engineering organization has two sub-teams: Development and QA. The Support organization has two sub-teams: Training and Services. And finally, the Marketing organization is divided into Sales and Advertising. The following image shoes the queue hierarchy for this example:
Each child queue is tied to its parent queue with the yarn.scheduler.capacity.<queue-path>.queues configuration property in the capacity-scheduler.xml file. The top-level "support", "engineering", and "marketing" queues would be tied to the "root" queue as follows:
Property: yarn.scheduler.capacity.root.queues
Value: support,engineering,marketing
Example:
<property> <name>yarn.scheduler.capacity.root.queues</name> <value>support,engineering,marketing</value> <description>The top-level queues below root.</description> </property>
Similarly, the children of the "support" queue would be defined as follows:
Property: yarn.scheduler.capacity.support.queues
Value: training,services
Example:
<property> <name>yarn.scheduler.capacity.support.queues</name> <value>training,services</value> <description>child queues under support</description> </property>
The children of the "engineering" queue would be defined as follows: Property: yarn.scheduler.capacity.engineering.queues Value: development,qa
Example:
<property> <name>yarn.scheduler.capacity.engineering.queues</name> <value>development,qa</value> <description>child queues under engineering</description> </property>
And the children of the "marketing" queue would be defined as follows: Property: yarn.scheduler.capacity.marketing.queues Value: sales,advertising
Example:
<property> <name>yarn.scheduler.capacity.marketing.queues</name> <value>sales,advertising</value> <description>child queues under marketing</description> </property>
