Cloudera Docs

Configuring cluster capacity with queues

You can manage your cluster capacity using queues to balance resource requirements of multiple applications from various users.

You can use the Capacity Scheduler to share cluster resources using FIFO (first in, first out) queues.

You can configure queues either by specifying the percentage of the capacity using the Relative mode or the actual units of vCores and memory using the Absolute mode or the fraction of the total capacity using the Weight mode. If you are upgrading your cluster, Weight mode is the default mode. If you are installing and configuring the cluster freshly, Relative mode is the default mode. You can change the allocation mode to the Absolute mode.

You can submit applications to different queues at multiple levels in the queue hierarchy if the capacity is available on the nodes in the cluster. Because total cluster capacity can vary, capacity configuration values are expressed using percentages, units, or fractions.

Example – Configuring capacity using the Relative mode

You can specify the capacity property to allocate a floating-point percentage value of cluster capacity to a queue. The following configurations divide the cluster resources between the "engineering", "support", and "marketing" organizations in a 6:1:3 ratio (60%, 10%, and 30%).

To specify the capacity property based on the above example, perform the following:

  1. In Cloudera Manager, select Clusters > YARN Queue Manager UI service. A graphical queue hierarchy is displayed in the Overview tab.
  2. Click on the three vertical dots on the root and select Edit Child Queues option.
  3. Enter the Configured Capacity of "engineering" to 60, "support" to 10, and "marketing" to 30.
  4. Click Save.

Example – Configuring capacity using the Absolute mode

You can specify the capacity in units of memory in MiB and the number of cores to a queue. If the total units of memory is 16384 MiB and 16 cores, and the the cluster resources between the "engineering", "support", and "marketing" organizations are allocated as follows:

Organization Memory vCores
engineering 9830 10
support 1638 2
marketing 4916 4

To specify the capacity property based on the above example, perform the following:

  1. In Cloudera Manager, select Clusters > YARN Queue Manager UI service. A graphical queue hierarchy is displayed in the Overview tab.
  2. Click on the three vertical dots on the root and select Edit Child Queues option.
  3. Enter the Configured Memory of "engineering" to 9830, "support" to 1638, and "marketing" to 4916.
  4. Enter the Configured vCores of "engineering" to 10, "support" to 2, and "marketing" to 4.
  5. Click Save.

Example – Configuring capacity using the Weight mode

You can specify the capacity in fractions of total resources. The resources are divided based on how the queue's weight relates to the sum of configured weights under the parent. The following configurations divide the cluster resources between the "engineering", "support", and "marketing" organizations in 6:1:3 (6/10, 1/10, and 3/10) fraction of total resources.

To specify the capacity property based on the above example, perform the following:

  1. In Cloudera Manager, select Clusters > YARN Queue Manager UI service. A graphical queue hierarchy is displayed in the Overview tab.
  2. Click on the three vertical dots on the root and select Edit Child Queues option.
  3. Enter the Configured Weight of "engineering" to 6, "support" to 1, and "marketing" to 3.
  4. Click Save.

Maximum Capacity: The Maximum Capacity is to the right field or fields after selecting Edit Child Queues in the Queue Manager UI for any queue. The name of the Maximum Capacity varies depending on the root queue property for Absolute, Relative or Weighted. The purpose of the Maximum Capacity is to allow queues to borrow resources from other queues when they are not in use and when they are suitable for the application and queue. A common use case for this is, a queue used for ad-hoc queries or low priority data sicence experiements which a user would like to borrow resources and are fine with loosing them at any point. Usually, it is a best practice to set the configured capacity equal to maximum capacity when a queue has interqueue preemption disabled otherwise the queue may tie up borrowed resources which are garunteed to another queue's configured capacity.