Cloudera Docs

Set user limits within a queue

Set a minimum percentage of resources allocated to each leaf queue user.

Minimum User Limit

The Minimum User Limit (minimum-user-limit-percent) property can be used to set the minimum percentage of resources allocated to each leaf queue user. The Minimum User Percentage is a soft limit on the smallest amount of resources a single user should get access to if they are requesting it. For example, if you set the Minimum User Percentage property to 10%, 10 users get 10% each, assuming they are all requesting it; this value is a soft limit because if one of the users asks for less capacity, you can place more users in the queue.

For example, to enable equal sharing of the services leaf queue capacity among five users, you must set the Minimum User Limit value to 20%.

To configure user limits based on this example, perform the following:

  1. In Cloudera Manager, select Clusters > YARN Queue Manager UI service. A graphical queue hierarchy is displayed in the Overview tab.
  2. Click on the three vertical dots on the Services queue and select View/Edit Queue Properties option.
  3. In the Queue Properties dialog-box, enter 20 in the Minimum User Limit text box.
  4. Click Save.

User Limit Factor

You can use the User Limit Factor (user-limit-factor) to control the maximum amount of resources that a single user can consume. User Limit Factor is set as a multiple of the queues minimum capacity where a user limit factor of one means the user can consume the entire minimum capacity of the queue. If the user limit factor is greater than 1, it is possible for the user consumption to grow into maximum capacity and if the value is set to less than 1, such as 0.5, a user can only obtain half of the minimum capacity of the queue. If you want a user to obtain the maximum capacity of a queue, set a value greater than 1 to allow the minimum capacity to be overtaken by a user that many times.
The following table shows how the queue resources are adjusted as users submit jobs to a queue with a minimum user limit percentage is set to 20%:
Table 1. Minimum User Limit
Minimum user limit percent = 20
1st user submits job Sole user gets 100% of the queue capacity
2nd user submits job Each user equally shares 50% of the queue capacity
3rd user submits job Each user equally shares 30.33% of the queue capacity
4th user submits job Each user equally shares 25% of the queue capacity
5th user submits job Each user equally shares 20% of the queue capacity
6th user submits job 6th user must wait for queue capacity to free up
  • Queue resources are adjusted in the same manner for a single user submitting multiple jobs in succession. If no other users are requesting queue resources, the first job receives 100% of the queue capacity. When the user submits a second job, each job receives 50% of queue capacity. When the user submits a third job, each job receives 33% of queue capacity. If a fourth user then submits a job, each job would receive 25% of queue capacity. When the number of jobs submitted by all users reaches a total of five, each job will receive 20% of queue capacity, and subsequent users must wait for queue capacity to free up (assuming preemption is not enabled).
  • The Capacity Scheduler also manages resources for decreasing numbers of users. As users’ applications finish running, other existing users with outstanding requirements begin to reclaim that share.
  • Note that despite this sharing among users, the FIFO application scheduling order of Capacity Scheduler does not change. This guarantees that users cannot monopolize queues by submitting new applications continuously. Applications (and thus the corresponding users) that are submitted first always get a higher priority than applications that are submitted later.

Capacity Scheduler’s leaf queues can also use the user-limit-factor property to control user resource allocations. This property denotes the fraction of queue capacity that any single user can consume up to a maximum value, regardless of whether or not there are idle resources in the cluster.

To configure the maximum limit (user-limit-factor) based on this example, perform the following:

  1. In Cloudera Manager, select Clusters > YARN Queue Manager UI service. A graphical queue hierarchy is displayed in the Overview tab.
  2. Click on the three vertical dots on the queue you want to set limit and select View/Edit Queue Properties option.
  3. In the Queue Properties dialog-box, enter 1 in the User Limit Factor text box.
  4. Click Save.

The default value of 1 means that any single user in the queue can, at the most, only occupy only the queue’s configured capacity. This prevents users in a single queue from monopolizing resources across all queues in a cluster. Setting the value to 2 restricts the queue's users to twice the queue’s configured capacity. Setting it to a value of 0.5 restricts any user from using resources beyond half of the queue capacity.