In order to use a Windows client to submit MapReduce jobs to an HDP 2.1 to a Linux cluster, take the following steps:
Deploy and incorporate the HDP 2.1 Hadoop client jar on the Windows client node. You can use the HDP 2.1 Hadoop client jars from the HDP 2.1 deployment. You do not need to use HDP 2.1 for Windows.
Set configurations in
mapred-site.xml
andyarn-site.xml
on the client node to describe how to interact with the HDP 2.1 Linux cluster.
To submit jobs from Windows clients, add/modify the following:
Set mapreduce.app-submission.cross-platform property in
mapred-site.xml
totrue
.Set mapreduce.application.classpath in the
mapred-site.xml
to use platform agnostic notations.Use {{ENV}} notation instead of %ENV% to expand the environment variables. For example, instead of
%HADOOP_MAPRED_HOME%/share/hadoop/mapreduce
use:{{HADOOP_MAPRED_HOME}}/share/hadoop/mapreduce
Set yarn.application.classpath property in
yarn-site.xml
use platform agnostic notations. Similarly use {{ENV}} notation instead of %ENV% to expand the environment variables.