Placement rules migration

Placement rules migration is part of the Fair Scheduler to Capacity Scheduler migration process. Learn about the limitations of this migration and how you can overcome them.

Migrating to CDP Private Cloud Base 7.1.5 or lower

Placement rules (used in Fair Scheduler) and mapping rules (used in Capacity Scheduler) are very different, therefore auto conversion of placement rules into mapping rules are not possible. You manually have to configure placement rules and mapping rules on the upgrade from CDH to CDP is completed.

The following are the most substantial limitation:

In Fair Scheduler you can use special placement rules like "default" or "specified" which are completely absent in Capacity Scheduler.
In Fair Scheduler you can set a "create" flag for every rule. Mapping rules do not support this.
In Fair Scheduler in case of nested rules the "create" flag is interpreted for both rules. This is not true in Capacity Scheduler.
If a rule can return a valid queue in Fair Scheduler, it proceeds to the next rule. Capacity Scheduler, on the other hand, returns “root.default”.

For more information see Fair Scheduler features and conversion details.

Migrating to CDP Private Cloud Base 7.1.6 or higher

In Cloudera Runtime 7.1.6 and later releases there is a new placement engine that supports a new JSON-based mapping rule format. These new mapping rules eliminated many previous mapping rule limitations caused by the transitioning from Fair Scheduler to Capacity Scheduler. Note that in weight mode more limitations are resolved than in percentage mode.

The new placement engine can be thought of as a superset of Fair Scheduler and Capacity Scheduler placement evaluation logic. This means two things:

Most items described in Fair Scheduler's <queuePlacementPolicy> section can be converted into Capacity Scheduler with some minor exceptions.
Full backward compatbility with the old queue-mapping rule format.

The following are the most substantial differences between the old placement and the new mapping rules:

Mapping rules can be created and managed using YARN Queue Manager.
The rules are described in JSON, however, this is transparent to the user in CDP. The generated JSON can be viewed in Cloudera Manager as part of the Capacity Scheduler Configuration Advanced Configuration Snippet (Safety Valve) setting.
You can configure what should happen when a rule cannot place the application to the target queue. There are three options: proceed to the next rule, reject the submission, place the application in the default queue. Previously Fair Scheduler proceeded to the next rule in such cases as it was more complicated to achieve a 'reject' or 'place to default queue' behavior.
New policies (mapping actions) are available: specified, defaultQueue, and reject.
The create flag was introduced: Non-existing queues are only created dynamically if it is enabled.

The following limitations remains when transitioning from the Fair Scheduler placement rules to the Capacity Scheduler mapping rules:

When using nested mapping rules, it is not possible to define two separate create flags for the parent and the leaf queue.
Fair Scheduler performs a strict validation whether a rule in the chain is reachable or not. The placement engine in Capacity Scheduler does not perform such a validation.

For more information, see Auto-converted Fair scheduler properties, Fair Scheduler features and conversion details, and Configuring Placement Rules.