These files are used to configure MapReduce jobs.
The following environment files define the log location for YARN and MRv2 for the daemons.
yarn-env.sh:export YARN_LOG_DIR=/var/log/hadoop-yarn/$USER
hadoop-env.sh:export HADOOP_LOG_DIR=/var/log/hadoop-mapred/$USER
Note | |
---|---|
|
Daemon .out Files
The log files with the .out extension for MRv2 and YARN daemons are located in
/var/log/hadoop-mapred/mapred
and in
/var/log/hadoop-yarn/yarn
. These files have the following
naming convention:
hadoop-mapred-historyserver-<HistoryServer_Host>.out
yarn-yarn-nodemanager-<nodemanager_host>.out
yarn-yarn-resourcemanager-<resourcemanager_host>.out
These .out files are created and written to during start-up of the MRv2 and YARN daemons. It is very rare that these files get populated, but they can be helpful when trying to determine why Resource Manager, Node Manager, or the Job History Server daemons are not starting up.
Daemon .log Files
The log files with the .log extension for MRv2 and YARN daemons are located in
/var/log/hadoop/mapred
and in
/var/log/hadoop-yarn/yarn
. These files have the following
naming convention:
hadoop-mapred-historyserver-<HistoryServer_Host>.log
yarn-yarn-nodemanager-<nodemanager_host>.log
yarn-yarn-resourcemanager-<resourcemanager_host>.log
These files show the log messages for the running daemons. If there are any errors encountered while the daemon is running, the stack trace of the error is logged in these files.
Note | |
---|---|
Log files are rotated daily by default, but can be adjusted by modifying the
|
Daemon .log.<date> Files
The .log.<date>files have the following format:
hadoop-mapred-historyserver-<HistoryServer_Host>.log.<date>
yarn-yarn-nodemanager-<nodemanager_host>.log.<date>
yarn-yarn-resourcemanager-<resourcemanager_host>.log.<date>
When .log files are rotated, the file name is appended with a date. An example of the file name would be:
mapred-mapred-historyserver-sandbox.log.2013-10-26
This indicates that the file was rotated on Oct 26, 2013. These files are useful when an issue has occurred multiple times, and comparing these older log files with the most recent log file can help uncover patterns of occurrence.
MapReduce V2 Container Log Files
The log files for specific applications and their Containers are located in HDFS. You can access Container log files using either the command line or the Resource Manager UI.
To access the Container log files using the command line, you first need to obtain the application ID by running the following command:
yarn application -list
This command returns a list of the applications, along with their application IDs:
yarn application -list 13/11/04 23:39:09 INFO client.RMProxy: Connecting to Resource Manager at sandbox/10.11.2.159:8050 Total number of applications (application-types: [] and states: [SUBMITTED, ACCEPTED, RUNNING]):1 Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL application_1383601692319_0008 QuasiMonteCarlo MAPREDUCE hdfs default ACCEPTED UNDEFINED 0% N/A
You can then use the application ID in the following command to access the Container log files:
yarn logs -applicationId application_1383601692319_0008
Note | |
---|---|
The log files are stored in HDFS under the following path:
|
Information about using the Resource Manager UI to access the Container log files is available on this page under "Using the Web GUI to Monitor Examples."