You must create the following HDFS directories after you upgrade:
YARN NodeManager remote applications log
HDFS Job History
To create the YARN NodeManager remote applications log
Open
/etc/hadoop/conf/yarn-site.xml
.Write down the value of the
yarn.nodemanager.remote-app-log-dir
so that you can use it in place of the${yarn.nodemanager.remote-app-log-dir}
variable in later examples. For example:${yarn.nodemanager.remote-app-log-dir} = /app-logs
Create the ${yarn.nodemanager.remote-app-log-dir} in HDFS.
hdfs dfs -mkdir ${yarn.nodemanager.remote-app-log-dir} hdfs dfs -chown -R yarn:hadoop ${yarn.nodemanager.remote-app-log-dir} hdfs dfs -chmod -R 777 ${yarn.nodemanager.remote-app-log-dir}
Create a JobHistory directory in HDFS.
Open
mapred-site.xml
.Write down the value of the
mapreduce.jobhistory.done-dir
so that you can use it in place of the${mapreduce.jobhistory.done-dir}
variable in later examples.Write down the value of the
mapreduce.jobhistory.intermediate-done-dir
so that you can use it in place of the${mapreduce.jobhistory.intermediate-done-dir}
variable in later examples.Create the JobHistory directories in HDFS.
hadoop dfs -mkdir ${mapreduce.jobhistory.done-dir} hadoop dfs -mkdir ${mapreduce.jobhistory.intermediate-done-dir} hadoop dfs -chown -R mapred:hadoop ${mapreduce.jobhistory.done-dir} hadoop dfs -chown -R mapred:hadoop ${mapreduce.jobhistory.intermediate-done-dir} hadoop dfs -chmod -R 777 ${mapreduce.jobhistory.done-dir} hadoop dfs -chmod -R 777 ${mapreduce.jobhistory.intermediate-done-dir}
Note You have to create the parent directories in HDFS by yourself. Grant the parent directories the same permissions.