What's New in Apache Hadoop YARN

Enhanced placement rules

A new placement rules evaluation engine is introduced to resolve some previous limitations. It supports a new JSON-based placement rules format. From Cloudera Runtime 7.2.9, the YARN Queue Manager UI supports only this new placement rules format. Legacy mapping rules are automatically converted into this new format when the YARN Queue Manager UI is used for the first time in Cloudera Runtime 7.2.9 or higher version.

With the new placement rules and evaluation engine, you can define an action that should happen if the target queue of the placement rule does not exist or it cannot be created. This action is called the fallback action and it can have the following values: Skip, PlaceDefault and Reject.

Placement rules can now be created and managed in a more user-friendly way in YARN Queue Manager using Placement rule policies. Placement rule policies provide shortcuts for the most common use cases of placement-rules. All legacy Capacity Scheduler mapping-rules are now available as these policies. They provide the same functionality without having to manually write rules. For example, a previous rule of the form u:%user:%primary_group.%user can now be achieved using the primaryGroupUser policy. If none of the predefined policies solve your use-case, a custom policy option is available that allows you to define the target queue manually.

For more information about placement rules, see Managing placement rules.

Dynamic child-queues in Weight mode

Support for dynamic child-queues is also introduced for the new Weight mode, continuing similar support in the previously existing relative & absolute modes. As before with other modes, dynamic child-queues for Weight mode can be enabled through the YARN Queue Manager UI.

Dynamic child-queues in the new Weight mode have additional enhancements. First, Weight mode supports two-levels of dynamic queues, while other modes only support one-level. And unlike in other modes where a parent-queue cannot have a mix of static and dynamic children, Weight mode supports parent-queues to have both dynamic and static child-queues simultaneously.

For more information, see Manage dynamic queues.

Weight mode for resource allocation

You can now allocate resources to queues using a new mode called Weight mode. This new mode lets you specify the capacity as weights/fractions of total resources instead of already existing ways of specifying it in percentages or in absolute numbers.

For more information about resource allocation, see Resource allocation overview.

Read-only access to Yarn Queue Manager UI for non-admins (Tech Preview)

You can now allow non-admin users to access YARN Queue Manager in a read-only mode. You can either create a new user account with read-only role or use any existing user account with read-only role in Cloudera Manager to access YARN Queue Manager UI. In the read-only access mode, the user can view all the configurations but cannot make any changes to the configurations.

For more information, see Provide read-only access to Queue Manager UI.

Fair sharing intra-queue preemption support

You can now enable intra-queue preemption for queues that are configured with fairness-based ordering. If the user has configured ordering-policies for queues (yarn.scheduler.capacity.<queue-path>.ordering-policy) to be Fair and if this new feature intra-queue preemption is enabled using YARN Queue Manager UI, then all the applications from the same user get a fair amount of resources. Thus, fair-ordering preemption now ensures that each application under a queue gets its fair-share, whether from a single user or several.

While that is for applications from a single user, for resource allocation across users, you can enforce a limit on each user’s resource-usage by setting the user limits (yarn.scheduler.capacity.<queue-path>.user-limit-factor).

For more information about ordering policies, see Set Ordering policies within a specific queue.

fs2cs conversion utility tool enhancements

The fs2cs conversion utility tool provides better scheduler transition for customers who are upgrading or migrating from one of the CDH releases.

The conversion from Fair Scheduler to Capacity Scheduler using the fs2cs conversion utility is now better through improved placement rules and a new placement rules evaluation engine (see below), a new Weight mode for capacity allocation (see below) with a new and more flexible auto queue creation feature for Capacity Scheduler.s

For more information, see Fair Scheduler to Capacity Scheduler transition.