Cloudera Docs

Placement rules evaluation engine

If you are upgrading to Cloudera Runtime 7.1.6 or higher you need to learn about the new placement rules evaluation engine.

This list is only an overview of the most prominent changes regarding placement rules evaluation. For a more detailed guide, see Managing placement rules.

JSON-based format

From Cloudera Runtime 7.1.6 placement rules are described in a JSON-based format. The new engine has full backward compatibility with the old format as well but only if you use safety valves which you must not do. Cloudera recommends managing your placement rules through the YARN Queue Manager UI. The format conversion is automatically done at the first time you use the YARN Queue Manager UI in CDP Private Cloud Base 7.1.6 or higher. For more information about the automatic placement rules format conversion, see Converting old mapping rule format to JSON-based placement rule format.

Enabling dynamic queues

The new placement rules evaluation engines support the create flag which was also introduced in Cloudera Runtime 7.1.6. Non-existing queues are only created if dynamic queue creation is enabled when the applicable placement rule is created. That means in order to enable dynamic child queues you have to perform two steps:
  1. Enable the dynamic auto child creation feature for the queue that you want to use as a parent queue for the dynamic child queues.
  2. Ensure that the Create the target queue if it does not exist? property is checked when the applicable placement rule is created.

Fallback action configuration

With the new placement rules and evaluation engine you can define the action that should happen if the target queue of a placement rule does not exist or it cannot be created. This action is called the fallback action. Fallback action can be configured for each placement rule by the user and it can have the following values:
  • Skip: Ignore the current rule and proceed to the next.
  • PlaceDefault: Place the application to the default queue root.default (unless it is overriden to something else). This is how Capacity Scheduler worked with the old mapping rules.
  • Reject: Reject the submission.

Placement rule policies

Placement rule policies were introduced as a more user friendly solution for mapping rule creation. Placement rule policies cover the functionality of the placement rules and provide shortcuts for the most common use cases.

All legacy Capacity Scheduler mapping rule options are now available as policy. They provide the same functionality without having to manually write rules. For example, the u:%user:%primary_group.%user rule can be achieved using the primaryGroupUser policy. However, a custom policy option is available that allows you to define the target queue manually without using any predefined placement rule policy.