What's New in Apache Hadoop YARN

Placement Rules

Placement Rules allow you to specify a set of rules for assigning jobs and applications to queues. You can now define placement rules to dynamically create queues based on the rules and the applications would be submitted to those queues. These predefined rules enable you to submit jobs without specifying the queue name at the time of job submission.

For more information about configuring placement rules, see Configure placement rules.

Data Locality

Capacity Scheduler leverages Delay Scheduling to honor task locality constraints. The scheduler counts the number of missed opportunities when the locality cannot be satisfied and waits for this count to reach a threshold before relaxing the locality constraint to the next level. You can configure this threshold using the YARN Queue Manager UI.

For more information about configuring data locality, see Configure data locality.

FPGA as a resource type

You can use FPGA as a resource type. For more information, see Use FPGA scheduling.

NodeManager Heartbeat

You can now control how many containers can be allocated in each NodeManager heartbeat. You can set the container assignments and off-switch assignments per NodeManager heartbeat.

For more information about configuring NodeManager heartbeat, see Configure NodeManager heartbeat.

Preemption

Preemption allows applications of higher priority to preempt applications of lower priority. If preemption is enabled, applications of higher priority do not have to wait because applications of lower priority have taken up the available capacity.

For more information on configuring preemption, see Configure preemption.

Same queue name under different hierarchies

Capacity Scheduler now support queues with the same name under different hierarchies, such us: root.a.q1 and root.b.q2.

Moving applications between queues (Technical Preview)

Capacity Scheduler now supports moving of application between queues.