What's New in Apache Hadoop YARN

Read-only access to YARN Queue Manager UI for non-admins

You can now allow non-admin users to access YARN Queue Manager in a read-only mode. You can either create a new user account with read-only role or use any existing user account with read-only role in Cloudera Manager to access YARN Queue Manager UI. In the read-only access mode, the user can view all the configurations but cannot make any changes to the configurations. Read-only mode will also take place during upgrades to YARN Queue Manager.

For more information, see Provide read-only access to Queue Manager UI.

Database requirement changes

After upgrading to Cloudera Data Platform (CDP) 7.1.9 or or CDP 7.1.9 cumalative hotfix (CHF) 1, you must migrate your YARN Queue Manager database to a PostgreSQL external database. This migration is needed because Queue Manager uses a different database as its backing store than what was used in previous versions of CDP 7.1.8 and below.

Fair sharing intra-queue preemption support

You can now enable intra-queue preemption for queues that are configured with fairness-based ordering. If the user has configured ordering-policies for queues (yarn.scheduler.capacity.<queue-path>.ordering-policy) to be Fair and if this new feature intra-queue preemption is enabled using YARN Queue Manager UI, then all the applications from the same user get a fair amount of resources. Thus, fair-ordering preemption now ensures that each application under a queue gets its fair-share, whether from a single user or several.

While that is for applications from a single user, for resource allocation across users, you can enforce a limit on each user’s resource-usage by setting the user limits (yarn.scheduler.capacity.<queue-path>.user-limit-factor).

For more information about ordering policies, see Set Ordering policies within a specific queue.

Mixed resource allocation mode (Technical Preview)

You can now specify the resources in mixed types. You can specify the actual units of vcores and memory resources for each queue or specify the percentage of the total resources used by each queue or specify a weight for each queue. You can use a combination of these allocation modes. The queues under one parent can also mix their modes.

For more information, see Mixed resource allocation mode.

YARN Queue Manager Maximum Memory value update

You are now able to configure absolute capacities beyond the current cluster capacity in YARN Queue Manager. You may exceed the Maximum Memory allowed for scenarios where the nodes will be added to the cluster manually or via autoscaling.

For more information about ordering policies, see Configuring Queue Manager for autoscaling.