Cloudera Docs

Set up queues

Capacity Scheduler queues can be set up in a hierarchy that reflects the database structure, resource requirements, and access restrictions required by the various organizations, groups, and users that utilize cluster resources.

The fundamental unit of scheduling in YARN is a queue. The capacity of each queue specifies the percentage of cluster resources that are available for applications submitted to the queue.

For example, suppose that a company has three organizations: Engineering, Support, and Marketing. The Engineering organization has two sub-teams: Development and QA. The Support organization has two sub-teams: Training and Services. And finally, the Marketing organization is divided into Sales and Advertising. The following image shows the queue hierarchy for this example:


Each child queue is tied to its parent queue with the yarn.scheduler.capacity.<queue-path>.queues configuration property in the capacity-scheduler.xml file. The top-level "support", "engineering", and "marketing" queues would be tied to the "root" queue.

To set the queues based on this example, perform the following:

  1. In Cloudera Manager, select the YARN service.
  2. Click the Configuration tab.
  3. Search for scheduler.
  4. In Capacity Scheduler Configuration Advanced Configuration Snippet (Safety Valve)add the following:
    • Add the following values for the root queue: 
      Name: yarn.scheduler.capacity.root.queues
Value: support,engineering,marketing
Description: The top-level queues below root.
    • Similarly, the children of the "support" queue would be defined as follows: 
      Name: yarn.scheduler.capacity.support.queues
Value: training,services
Description: child queues under support
    • The children of the "engineering" queue would be defined as follows:
      Name: yarn.scheduler.capacity.engineering.queues 
Value: development,qa 
Description: child queues under engineering
    • And the children of the "marketing" queue would be defined as follows:
      Name: yarn.scheduler.capacity.marketing.queues
Value: sales,advertising 
Description: child queues under marketing