Cloudera Docs

Placement rule policies

When creating a placement rule you have to set its policy to define the logic that is taken into account when specifying which queue should be used for a submitted job. There are many predefined policies you can choose from, or you can create a custom policy.

The following table lists the names of the policies, the options displayed in the Placement Rule creation dialog box in the Queue Manager UI, and their detailed description:

Table 1. Placement rule policies
Policy Queue Manager UI Description
user

Place the application into a queue named for the user.

Places the application into a queue which matches the username of the submitter.
primaryGroup

Place the application into a queue named for the user’s primary group.

Places the application into a queue which matches the primary group of the submitter.
primaryGroupUser

Place the application into a queue named for the user that is the child of a queue named for the user’s primary group.

Places the application into the queue hierarchy [parentQueue].<primaryGroup>.<userName>. The parentQueue is optional.
secondaryGroup

Place the application into a queue named for the user’s secondary group.

Places the application into a queue which matches the secondary group of the submitter.
secondaryGroupUser

Place the application into a queue named for the user that is the child of a queue named for the user’s secondary group.

Places the application into the queue hierarchy [parentQueue].<secondaryGroup>.<userName>. The parentQueue is optional.
applicationName

Place the application into a queue named for the application.

Places the application into a queue which matches the name of the application.

specified

Place the application in the queue specified at the run time.

Places the application to the queue that was defined during submission.
reject

Reject the application.

Rejects the submission.
defaultQueue

Place the application into the default queue.

Places the application into the default queue root.default or to its overwritten value.
setDefaultQueue

Set the default queue to:

Changes the default queue from root.default.

This policy does not permanently change the default queue. It is always root.default when the evaluation begins when an application is submitted.

However, the adjusted default queue remains in effect until the placement rules evaluation is completed.
custom

Use the following custom policy:

Enables the user to use custom placement strings.

Using the custom placement policy

The custom placement policy can describe other policies with the appropriate variable placeholders. For example, primaryGroupUser with the parent queue root.groups can be expressed as: root.groups.%primary_group.%user

The Custom policy variables table described what variables are available if you intend to use the custom policy.

Internally, the tool populates certain variables with appropriate values. These can be used if custom mapping policy is selected. The placement rules evaluation engine does only minimal verification when it comes to replacing them. Therefore it is your responsibility to provide the correct string.

Table 2. Custom policy variables
Variable Meaning
%application

The name of the submitted application.
%default

The default queue of the scheduler.
%primary_group

Primary group of the submitter.
%secondary_group

Secondary (supplementary) group of the submitter.
%specified

Contains the queue that the submitter defined.

It is a standalone variable, do not combine it with other custom variables or paths.

If the specified target queue is default this variable will not be set. If the target queue is the default queue, it should be specified with the root.default parent path.
%user

The user who submitted the application.

If you submit a MapReduce application to a queue root.users.mrjobs, the value of %specified will be set to root.users.mrjobs.

Few predefined policies can be achieved with the custom policy. For example, instead of using the specified policy, you can use the custom policy and set the customPlacement field to %specified.

However, you have more control over the policy if you use custom instead of the predefined one. That is because you can also append or prepend an extra string to these variables. As a result, for example, the following setting is possible:root.groups.%primary_group.admins.%user.longrunningjobs

The string must be resolved to a valid queue path to have a proper placement.

Differences between legacy modes and weight mode

There are some cases when the legacy resource allocation modes (absolute and relative mode) behave differently than weight mode.

The create flag
  • Legacy modes: It has no effect if the parent is not managed.
  • Weight mode: It applies to all parent queues. However, if yarn.scheduler.capacity.<queue-path>.auto-queue-creation-v2.enabled is set to false, then the queue will not be created.
The nested rules primaryGroupUser and secondaryGroupUser
  • Legacy modes: They expect the parent queues to exist, meaning they cannot be created automatically. More specifically, when you use primaryGroupUser it will result in a queue path similar to root.<primaryGroup>.<userName> and the root.<primaryGroup> parent queue must exist. It can be a managed parent in order to have the userName leaf created automatically, but the parent still has to be created manually.
  • Weight mode: As long as the parent allows dynamic queues to be created, there is no limitations. The requested queues will be created.