What's New in Apache Hadoop YARN
Learn about the new features of Hadoop YARN in Cloudera Runtime 7.2.11.
Migrating database configuration to a new location
The operations performed on queues in Queue Manager UI are stored as Queue Manager versions. You can either store these versions in the default database location on the host or configure a new location using Cloudera Manager UI. For security reasons, if you do not want to allow users to access the default database, you can move the database file to an alternative location. During an upgrade, you can move the database file to some other location and then restore this file to the default location after the upgrade.
For more information, see Migrating database configuration to a new location.
Queue Manager REST API
You can now use the Queue Manager REST APIs to automate the Queue Manager operations using the HTTP endpoints. The Queue Manager REST server exposes endpoints that provide the start, stop, add, delete, change queue capacity, and change queue properties operations.
For more information, see Configuring and using Queue Manager REST API.
Read-only access to Yarn Queue Manager UI for non-admins
You can now allow non-admin users to access YARN Queue Manager in a read-only mode. You can either create a new user account with read-only role or use any existing user account with read-only role in Cloudera Manager to access YARN Queue Manager UI. In the read-only access mode, the user can view all the configurations but cannot make any changes to the configurations.
For more information, see Provide read-only access to Queue Manager UI.
Fair sharing intra-queue preemption support
You can now enable intra-queue preemption for queues that are configured with fairness-based ordering. If the user has configured ordering-policies for queues (yarn.scheduler.capacity.<queue-path>.ordering-policy) to be Fair and if this new feature intra-queue preemption is enabled using YARN Queue Manager UI, then all the applications from the same user get a fair amount of resources. Thus, fair-ordering preemption now ensures that each application under a queue gets its fair-share, whether from a single user or several.
While that is for applications from a single user, for resource allocation across users, you can enforce a limit on each user’s resource-usage by setting the user limits (yarn.scheduler.capacity.<queue-path>.user-limit-factor).
For more information about ordering policies, see Set Ordering policies within a specific queue.
Lazy Preemption
You can now enable the LazyPreemption feature to optimize the selection of containers from queues for preemption. Containers are preempted only if the resources can be used by another application. The LazyPreemption feature makes resource preemption more efficient. It prevents excessive preemption and ensures that container selection is continuously optimized. LazyPreemption avoids scenarios when preempted resources cannot be used by the application that needs the resource, and then such preempted resources go back to the over-utilized queue again.
For more information, see Enabling LazyPreemption.
Set default Partition Expression
You can set a default partition on a queue. The default partition is used if no partition is specified when the application is submitted to the queue. By default, this field is empty, so applications get containers from nodes without a partition or label.
For more information, see Setting a default partition expression.
Dynamic Queue Scheduling (Tech Preview)
The Dynamic Queue Scheduling feature enables you to dynamically change queue resource allocation. Currently, the feature is only supported in relative resource allocation mode.
For more information, see Dynamic Queue Scheduling.
Auto Queue Creation Template (Technical Preview)
You can set the minimum and maximum capacities using the Auto Queue Creation template. These capacities will be applied for every dynamic child queue under that particular parent queue.
For more information, see Enable dynamic auto child creation in weight mode.
Dynamic Child Creation Improvements
- You can now enable dynamic child creation for all queues. Previously, there must be at least one static child queue under the parent queue for which you want to enable this feature.
- Auto queue creation depth limit - Auto queue creation depth limit limits the number of queue levels from a parent queue. While previously auto queue creation depth limit limited the levels of queue path relative to its first existing parent queue, now it limits the level of queue path relative to its first static parent queue.
- The Auto Queue Deletion feature - Dynamic queues are deleted automatically after 300 seconds of the last job finished on them. This feature is enabled by default but supported only in weight allocation mode. You can prevent the automatic deletion of dynamic queues by disabling the Auto Queue Deletion feature in Cloudera Manager. For more information, see Disabling auto queue deletion.
