Cloudera Docs

Configure cluster capacity with queues

You can manage your cluster capacity using queues to balance resource requirements of multiple applications from various users.

You can use the Capacity Scheduler to share cluster resources using FIFO (first in, first out) queues.

You can configure queues either to own a fraction of the capacity using the Relative mode, or the actual units of vCores and memory using the Absolute mode. The Relative allocation mode is the default. You can change the allocation mode to the Absolute mode.

You can submit applications to different queues at multiple levels in the queue hierarchy if the capacity is available on the nodes in the cluster. Because total cluster capacity can vary, capacity configuration values are expressed using percentages or units.

Example – Configuring capacity using the Relative mode

You can specify the capacity property to allocate a floating-point percentage value of cluster capacity to a queue. The following configurations divide the cluster resources between the "engineering", "support", and "marketing" organizations in a 6:1:3 ratio (60%, 10%, and 30%).

To specify the capacity property based on the above example, perform the following:

  1. In Cloudera Manager, select Clusters > YARN Queue Manager UI service. A graphical queue hierarchy is displayed in the Overview tab.
  2. Click on the three vertical dots on the root and select Edit Child Queues option.
  3. Enter the Configured Capacity of "engineering" to 60, "support" to 10, and "marketing" to 30.
  4. Click Save.

Example – Configuring capacity using the Absolute mode

You can specify the capacity in units of memory in MiB and the number of cores to a queue. If the total units of memory is 16384 MiB and 16 cores, and the the cluster resources between the "engineering", "support", and "marketing" organizations are allocated as follows:

Organization Memory vCores
engineering 9830 10
support 1638 2
marketing 4916 4

To specify the capacity property based on the above example, perform the following:

  1. In Cloudera Manager, select Clusters > YARN Queue Manager UI service. A graphical queue hierarchy is displayed in the Overview tab.
  2. Click on the three vertical dots on the root and select Edit Child Queues option.
  3. Enter the Configured Memory of "engineering" to 9830, "support" to 1638, and "marketing" to 4916.
  4. Enter the Configured vCores of "engineering" to 10, "support" to 2, and "marketing" to 4.
  5. Click Save.