Cloudera Docs

Placement Rules transition

Placement rules transition is part of the Fair Scheduler to Capacity Scheduler transition process. Learn about the limitations of this transition and how you can overcome them.

Placement rules (used in Fair Scheduler) and mapping rules (used in Capacity Scheduler) are very different, therefore auto conversion of placement rules into mapping rules are not possible. You manually have to configure placement rules and mapping rules on the upgrade from CDH to CDP is completed.

The following are the most substantial limitation:

  • In Fair Scheduler you can use special placement rules like "default" or "specified" which are completely absent in Capacity Scheduler.

  • In Fair Scheduler you can set a "create" flag for every rule. Mapping rules do not support this.

  • In Fair Scheduler in case of nested rules the "create" flag is interpreted for both rules. This is not true in Capacity Scheduler.

  • If a rule can return a valid queue in Fair Scheduler, it proceeds to the next rule. Capacity Scheduler, on the other hand, returns “root.default”.

For more information see Fair Scheduler features and conversion details.

In Cloudera Runtime 7.1.6 and later releases there is a new placement engine that supports a new JSON-based placement rule format. These new placement rules eliminated many previous placement rules limitations caused by the transitioning from Fair Scheduler to Capacity Scheduler. Note that in weight mode more limitations are resolved than in percentage mode.

The new placement engine can be thought of as a superset of Fair Scheduler and Capacity Scheduler placement evaluation logic. This means two things:
  • Everything that could be described in Fair Scheduler's <queuePlacementPolicy> section can be converted into Capacity Scheduler with some minor exceptions.
  • Full backward compatbility with the old queue-mapping rule format.
The following are the most substantial differences between the old placement and the new placement rules:
  • The rules are described in JSON, however, this is transparent to the user in CDP. The generated JSON can be viewed in Cloudera Manager as part of the Capacity Scheduler Configuration Advanced Configuration Snippet (Safety Valve) setting.
  • You can configure what should happen when a rule cannot place the application to the target queue. There are three options: proceed ti the next rule, reject the submission, place the application in the default queue.
  • New policies (mapping actions) are available: specified, defaultQueue, and reject.
  • The create flag was introduced: Non-existing queues are only created dynamically if it is enabled.
The following limitations remains when transitioning from the Fair Scheduler placement rules to the Capacity Scheduler placement rules:
  • When using nested placement rules, it is not possible to define two separate create flag.
  • Fair Scheduler performs a strict validation whether a rule in the chain is reachable or not. The placement engine in Capacity Scheduler does not perform such a validation.

For more information, see Auto-converted Fair scheduler properties, Fair Scheduler features and conversion details, and Managing placement rules.

The following table shows how the fs2cs conversion utility converts the olda placement rules into the new JSON-based placement rule format. This conversion happens automatically.
Table 1. Automatic placement rule conversion
Fait Scheduler placement rule JSON-based Capacity Scheduler placement rules 
<rule name=”specified” />
 
{
  "type":"user",
  "matches":"*",
  "policy":"specified",
  "fallbackResult":"skip"
}

<rule name=”user” />
 
{
  "type":"user",
  "matches":"*",
  "policy":"user",
  "fallbackResult":"skip"
}

<rule name=”default” />
 
{
  "type":"user",
  "matches":"*",
  "policy":"defaultQueue",
  "fallbackResult":"skip"
}

<rule name=”default” queue=”root.tmp” />
 
{
  "type":"user",
  "matches":"*",
  "policy":"setDefaultQueue",
  “value”: “root.tmp”,
  "fallbackResult":"skip"
},
{
  "type":"user",
  "matches":"*",
  "policy":"defaultQueue",
  "fallbackResult":"skip"
}

or

{
  "type":"user",
  "matches":"*",
  "policy":"custom",
  “customPlacement:”root.tmp”,
  "fallbackResult":"skip"
}

<rule name=”primaryGroup” />
 
{
  "type":"user",
  "matches":"*",
  "policy":"primaryGroup",
  "fallbackResult":"skip"
}

<rule name=”secondaryGroupExistingQueue” />
 
{
  "type":"user",
  "matches":"*",
  "policy":"secondaryGroup",
  "fallbackResult":"skip"
}

<rule name="nestedUserQueue">
    <rule name="secondaryGroupExistingQueue" />    </rule>

{
  "type":"user",
  "matches":"*",
  "policy":"secondaryGroupUser",
  "fallbackResult":"skip"
}

<rule name="nestedUserQueue">
    <rule name="primaryGroup" />
</rule>

{
  "type":"user",
  "matches":"*",
  "policy":"primaryGroupUser",
  "fallbackResult":"skip"
}

<rule name="nestedUserQueue">
    <rule name="default" queue=”root.users” />
</rule>

{
  "type":"user",
  "matches":"*",
  “parentQueue”: “root.users”,
  "policy":"user",
  "fallbackResult":"skip"
}

<rule name=”reject” />
 
{
  "type":"user",
  "matches":"*",
  "policy":"reject",
  "fallbackResult":"skip"
}