3.2.5. Log Files

These files are used to configure MapReduce jobs.

The following environment files define the log location for YARN and MRv2 for the daemons.

Note

The Job History Server runs as the “mapred” user, and it will use the hadoop-env.sh file. The Resource Manager and the Node Manager run as the “yarn” user, and they will use the yarn-env.sh file. The Application Master and Container log files are removed from the local directories once the Job finishes, and are moved to the HDFS2 directory. The Job History Server will display the Application Master and Containers log files, which are stored in HDFS2 for 30 days.

Daemon .out Files

The log files with the .out extension for MRv2 and YARN daemons are located in /var/log/hadoop-mapred/mapred and in /var/log/hadoop-yarn/yarn. These files have the following naming convention:

These .out files are created and written to during start-up of the MRv2 and YARN daemons. It is very rare that these files get populated, but they can be helpful when trying to determine why Resource Manager, Node Manager, or the Job History Server daemons are not starting up.

Daemon .log Files

The log files with the .log extension for MRv2 and YARN daemons are located in /var/log/hadoop/mapred and in /var/log/hadoop-yarn/yarn. These files have the following naming convention:

These files show the log messages for the running daemons. If there are any errors encountered while the daemon is running, the stack trace of the error is logged in these files.  

	Note
	Log files are rotated daily by default, but can be adjusted by modifying the /etc/hadoop/conf/log4j.properties file.

Daemon .log.<date> Files

The .log.<date>files have the following format:

When .log files are rotated, the file name is appended with a date. An example of the file name would be:

mapred-mapred-historyserver-sandbox.log.2013-10-26

This indicates that the file was rotated on Oct 26, 2013. These files are useful when an issue has occurred multiple times, and comparing these older log files with the most recent log file can help uncover patterns of occurrence.

MapReduce V2 Container Log Files

The log files for specific applications and their Containers are located in HDFS. You can access Container log files using either the command line or the Resource Manager UI.

To access the Container log files using the command line, you first need to obtain the application ID by running the following command:

yarn application -list

This command returns a list of the applications, along with their application IDs:

yarn application -list
13/11/04 23:39:09 INFO client.RMProxy: Connecting to Resource Manager at sandbox/10.11.2.159:8050
Total number of applications (application-types: [] and states: [SUBMITTED, ACCEPTED, RUNNING]):1
                Application-Id	    Application-Name	    Application-Type	      User	     Queue	             State	       Final-State	       Progress	                       Tracking-URL
application_1383601692319_0008	     QuasiMonteCarlo	           MAPREDUCE	      hdfs	   default	          ACCEPTED	         UNDEFINED	             0%	                                N/A

You can then use the application ID in the following command to access the Container log files:

yarn logs -applicationId application_1383601692319_0008

	Note
	The log files are stored in HDFS under the following path: /app-logs/hdfs/logs/<application_id>/<hostnames*>_4545

Information about using the Resource Manager UI to access the Container log files is available on this page under "Using the Web GUI to Monitor Examples."