Scheduler transition limitations
There are some hard limitations on converting a Fair Scheduler configuration into a Capacity Scheduler configuration as these two schedulers are not equivalent. Learning about these major limitations can help you understand the challenges you might encounter after the scheduler transitions.
Static and dynamic leaf queues cannot be created on the same level
If you have a parent queue defined in
capacity-scheduler.xml file with at
least a single leaf queue, it is not possible to dynamically create a new leaf under this
Resolved: Weight mode and Dynamic Auto Child Creation is supported from Cloudera Runtime 7.1.6. In weight mode there can be static and dynamic leaf queues on the same level.
Placement rules and mapping rules are different
Depending on your target cluster version you will encounter different placement rules limitations. For more details, see Placement rules transition.
The capacity value of dynamic queues is fixed
In Fair Scheduler, fair shares are recalculated each time a new queue is created. In contrast, Capacity Scheduler assigns a predefined percentage value for dynamically created queues.
This predefined percentage can be changed, but it is fixed until the scheduler is reconfigured. Once this value reaches 100, the next dynamic queue will be created with the value 0. For example, if the value is set to 25.00, then the fifth queue under the same parent will have a capacity of 0.
Capacity Scheduler equivalent (capacity)
In Cloudera Runtime 7.1.5 and lower versions the fs2cs conversion utility ensures that all percentages of direct children under one parent queue add up exactly to 100.000%, as it is demonstrated in the table. For example, all queues under root.users: root.users.alice + root.users.bob + root.users.charlie = 100.000%.
Weights are converted into percentage-based capacities the following way: On queue-level root, there are 2 queues: default and users. Because it is specified as 10 + 30 weights (40 altogether), 1 “unit of weight” is 2.5%. This is why root.default has 25% and root.users has 75% of the capacity. This calculation can be applied to all queue-levels.
Resolved: Weight mode and Dynamic Auto Child Creation is supported from Cloudera Runtime 7.1.6. and the fs2cs conversion utility converts into weight mode by default.