What's New in YARN and YARN Queue Manager

Cloudera Runtime 7.3.2

Hadoop rebase summary

In Cloudera Runtime 7.3.2, Apache Hadoop is rebased to version 3.4.1. The Apache Hadoop upgrade improves overall performance and includes all the new features, improvements, and bug fixes from versions 3.2, 3.3, and 3.4.

Table 1. New features added from Apache Hadoop 3.2 to 3.4 versions

Apache Hadoop version	Apache Jira	Name	Description
3.3	YARN-8762	Interactive shell support in Hadoop YARN 3.3.0	Hadoop YARN 3.3.0 now incorporates the Interactive Shell for container execution. This feature provides the following capabilities: Enables you to run commands and conduct real-time debugging within both standard YARN and Docker containers. Provides access through a new Command Line Interface (CLI) supporting the sh shell or using a dedicated terminal link within the new YARN UI2. Ensures that only the container owner is authorized to access the shell using stringent authentication protocols. Allows administrators to deactivate this feature using a specific configuration flag.

Table 2. Improvements added from Apache Hadoop 3.2 to 3.4 versions

Apache Hadoop version	Apache Jira	Name	Description
3.2	MAPREDUCE-7069	Added ability to specify user environment variables individually	Environment variables can now be specified for MapReduce tasks using individual properties. The following formats are used for specifying the environment variables: mapreduce.map.env.VARNAME=value mapreduce.reduce.env.VARNAME=value yarn.app.mapreduce.am.env.VARNAME=value yarn.app.mapreduce.am.admin.user.env.VARNAME=value
3.3	YARN-8489	YARN Service Status Based on Dominant Component	The YARN service status reporting is now based on the status of the dominant component, improving the existing termination policies, for example, ALWAYS, NEVER. The final status of the dominant component determines the final state of the job.
3.4	YARN-9279	YARN Hamlet Package Removal	The deprecated org.apache.hadoop.yarn.webapp.hamlet package is now completely removed to improve maintainability. This is an incompatible change in Hadoop YARN 3.4.0+. Applications relying on this old package must be updated to use the org.apache.hadoop.yarn.webapp.hamlet2 package. This affects the YARN webapp component.
3.4	YARN-10820	Enhanced Reliability for YARN node list Command	The thread-safety issue is fixed in GetClusterNodesRequestPBImpl, that previously caused intermittent failures, such as java.lang.ArrayIndexOutOfBoundsException, with the YARN node list command. This change affects the YARN client in Hadoop YARN 3.4.0, 3.3.2, and 3.2.4, thereby, eliminating random crashes when running the YARN node list command.

Table 3. Issues fixed between Apache Hadoop versions 3.2 to 3.4

Apache Hadoop version	Apache Jira	Name	Description
3.3	MAPREDUCE-6190	MapReduce task initialization Timeout issue	Previously, MapReduce jobs stopped responding if a task terminated before sending its first heartbeat, as the task never timed out and remained stuck indefinitely in a "STARTING" state. This issue is now resolved by introducing a dedicated timeout mechanism specifically designed to catch and terminate tasks that fail to initialize and send their first heartbeat.
3.4	YARN-9809	Miscommunication between RM and NM when NodeManagers are unhealthy	Previously, if a NodeManager (NM) was registered in an unhealthy state, it did not communicate the status immediately. As a result, the Resource Manager (RM) mistakenly scheduled many containers to that unhealthy node before the first heartbeat was received. Once the first heartbeat finally arrived, the RM recognize the unhealthy status and abruptly ended all the recently scheduled containers, causing unnecessary task failures and wasted resources. This issue is now resolved and NMs now explicitly supply their health status during their initial registration with the RM.