Cloudera Docs

Setting Maximum Parallel Application

You can set the maximum number of applications that can run at the same time in a cluster. Ensure you understand how the parallel application limits are inherited from the parent queue in the queue hierarchy before using this option.

You can set parallel application limits for all queues, all users, and at the user level. When the maximum parallel applications limit is reached, application submissions are not rejected and instead they stay in the ACCEPTED state until they are eligible to run.

The evaluation of the limits happens in the following order:

  1. Maximum Applications - if the limit is exceeded, the submission is rejected immediately.
  2. Maximum Parallel Applications - if the limit is exceeded, the submission is accepted, but the application does not transition to the RUNNING state. It stays in the ACCEPTED state until the queue or user limits are satisfied.
  3. Maximum AM Resource Limit - if there are too many Application Masters running, the application stays in the ACCEPTED state until the resources become available.

Example - Application Limits

The maximum parallel application limit is an inherited property in the queue hierarchy. You can lower the limit down the hierarchy, but setting a higher value than its higher queue in the queue structure, the limit will not be effective. The following example provides information about how the parallel application limit is applied in a queue hierarchy. The Maximum Parallel Applications: All Queues value is set to 250 at the scheduler level and 500 is set at the root level.

Example queue hierarchy:

  • Maximum 250 applications can run parallel under the root.a queues.
  • Maximum 50 applications can run parallel under the root.a.a1 queues.
  • Maximum 10 applications can run parallel under the root.a.a1.a2 queues.
  • Maximum 10 applications can run parallel under the root.a1.a2.a3 queues. Even though the Maximum Parallel Applications limit is not set for .a3, the default value of 250 applies for that queue; however, the parent queue (root.a.a1.a2) has a lower value of 10, and the child .a3 queue cannot exceed that limit.
  • Maximum 10 applications can run parallel under the root.a1.a2.a3.a4 queue. Even though the queue level limit is configured for 15 applications, the parents restrict this limit to 10.
  • Maximum 50 applications can run parallel under the root.a.b queue.

Example - User Limits

  • Scenario 1 - In this scenario, there are two users, User1 and User2. When you set the Maximum Parallel Applications: All Queues to 100 and Maximum Parallel Applications: All Users to 50, both User1 and User2 can have 50 applications running in parallel.
  • Scenario 2 - In this scenario, there are four users, User1, User2, User3, and User4. You can set the limits as follows:

    • Maximum Parallel Applications: All Queues - 100

    • Maximum Parallel Applications: All Users - 15

    • Maximum Parallel Applications: Per Users

      • User1 - 10

      • User 2 - 25

    User1 can have 10 applications running in parallel. User2 can have 25 applications running in parallel. User3 and User4 can have 15 applications running in parallel.
  • Scenario 3: In this scenario, the Maximum Parallel Applications limit for root.a is set to 5. The Maximum Parallel Applications: Per Users for User1 is set to 10. Even though User1 can run 10 parallel applications, in the root.a queue, User1 can run only 5 maximum applications. User1 can also run 5 more applications on some other queue.
  • Scenario 4: In this scenario, the Maximum Parallel Applications limit for root.b is set to 20. The Maximum Parallel Applications: Per Users limit for User2 is set to 10. User2 can run 10 parallel applications, in the root.b queue. Some other users can run 10 more applications in the root.b queue.