List of Pages in Category YARN (31 pages)
YARN (Yet Another Resource Negotiator)
A general architecture for running distributed applications. YARN specifies the following components:
- ResourceManager - A master daemon that authorizes submitted jobs to run, assigns an ApplicationMaster to them, and enforces resource limits.
- ApplicationMaster - A supervisory task that requests the resources needed for executor tasks. An ApplicationMaster runs on a different NodeManager for each application. The ApplicationMaster requests containers, which are sized by the resources a task requires to run.
- NodeManager - A worker daemon that launches and monitors the ApplicationMaster and task containers.
- JobHistory Server - Keeps track of completed applications.
The ApplicationMaster negotiates with the ResourceManager for cluster resources—described in terms of a number of containers, each with a certain memory limit—and then runs application-specific processes in those containers. The containers are overseen by NodeManagers running on cluster nodes, which ensure that the application does not use more resources than it has been allocated.
MapReduce v2 (MRv2) is implemented as a YARN application.
*
C
- Cluster Utilization Reports
- Configuring ADLS Gen1 Connectivity
- Configuring ADLS Gen2 Connectivity
- Configuring Oozie to Enable MapReduce Jobs To Read/Write from Amazon S3
- Configuring Oozie to Enable MapReduce Jobs To Read/Write from Microsoft Azure (ADLS)
- Configuring the Fair Scheduler
- Configuring TLS/SSL for HDFS, YARN and MapReduce
- ContainerExecutor Error Codes (YARN)
- Creating a Custom Cluster Utilization Report
)D
E
M
- Managing the Sqoop 1 Client
- Managing YARN
- Managing YARN (MRv2) and MapReduce (MRv1)
- MapReduce (MRv1) and YARN (MRv2) High Availability
- Monitoring YARN Applications
N
R
- REQUEST_POOL Query Option
- Resource Management
- ResourceManager Metrics
- Running Spark Applications on YARN
S
T
U
W
Y