YARN Resource Management
Also available as:
PDF
loading table of contents...

Contents

1. Capacity Scheduler
Enabling Capacity Scheduler
Setting up Queues
Hierarchical Queue Characteristics
Scheduling Among Queues
Controlling Access to Queues with ACLs
Default Queue Mapping based on User or Group
Configuring Queue Mapping for Users and Groups to Specific Queues
Configuring Queue Mapping for Users and Groups to Queues with the Same Name
Enabling Override of Default Queue Mappings
Managing Cluster Capacity with Queues
Resource Distribution Workflow
Resource Distribution Workflow Example
Setting User Limits
Application Reservations
Using Flexible Scheduling Policies
FIFO vs. Fair Sharing
Configuring Queue Ordering Policies
Best Practices for Ordering Policies
Starting and Stopping Queues
Setting Application Limits
Preemption
Preemption Workflow
Preemption Configuration
Scheduler User Interface
2. CGroups
Enabling CGroups
Using CGroups
3. Using CPU Scheduling to Allocate Resources
About YARN Resource Allocation
Configuring CPU Scheduling
Using CPU Scheduling
4. Log Aggregation for Long-running Applications
Enable Log Aggregation
Including and Excluding Log Files in YARN Applications Running on Slider
5. Node Labels
Configuring Node Labels
Using Node Labels
6. Running Applications on YARN Using Slider
System Requirements
Operating System Requirements
Installing Apache Slider
Additional Configuration Steps for Secure Clusters
Running Applications on Slider
The Slider Application Package
Install the Application Package
Start the Application
Verify the Application
Manage the Application Lifecycle
The Application Registry
Running HBase on YARN via Slider
Downloading and Installing the HBase Application Package
Configuring HBase on YARN via Slider
Configuring HBase on YARN on Secure Clusters
Components in HBase on YARN
Launching an HBase Application Instance
Deployment Considerations
Running Accumulo on YARN via Slider
Downloading and Installing the Accumulo Application Package
Configuring Accumulo on YARN
Configuring Accumulo on YARN on Secure Clusters
Launching an Accumulo Application Instance
Client Connections to Accumulo and Retrieving Effective accumulo-site.xml
Deployment Considerations
7. Running Multiple MapReduce Versions Using the YARN Distributed Cache
8. Timeline Server
Timeline Server 1.5 Overview
Upgrading Timeline Server 1.0 to 1.5
Configuring the Timeline Server
Enabling Generic Data Collection
Configuring Per-Framework Data Collection
Configuring the Timeline Server Store
Configuring Timeline Server Security
Running the Timeline Server
Accessing Generic Data from the Command- Line
Publishing Per-Framework Data in Applications
9. Using the YARN REST APIs to Manage Applications
10. Work-Preserving Restart
Configuring the ResourceManager for Work-Preserving Restart
Configuring NodeManagers for Work-Preserving Restart
11. Using the YARN CLI to View Logs for Running Applications
View all Log Files for a Running Application
View a Specific Log Type for a Running Application
View ApplicationMaster Log Files
List Container IDs
View Log Files for One Container
Show Container Log File Information
View a Portion of the Log Files for One Container
Download Logs for a Running Application
Display Help for YARN Logs