3.1.6. Using a Windows Client to Submit MapReduce Jobs

In order to use a Windows client to submit MapReduce jobs to an HDP 2.1 to a Linux cluster, take the following steps:

  • Deploy and incorporate the HDP 2.1 Hadoop client jar on the Windows client node. You can use the HDP 2.1 Hadoop client jars from the HDP 2.1 deployment. You do not need to use HDP 2.1 for Windows.

  • Set configurations in mapred-site.xml and yarn-site.xml on the client node to describe how to interact with the HDP 2.1 Linux cluster.

To submit jobs from Windows clients, add/modify the following:

  • Set mapreduce.app-submission.cross-platform property in mapred-site.xml to true.

  • Set mapreduce.application.classpath in the mapred-site.xml to use platform agnostic notations.

  • Use {{ENV}} notation instead of %ENV% to expand the environment variables. For example, instead of %HADOOP_MAPRED_HOME%/share/hadoop/mapreduce use:

     {{HADOOP_MAPRED_HOME}}/share/hadoop/mapreduce   
  • Set yarn.application.classpath property in yarn-site.xml use platform agnostic notations. Similarly use {{ENV}} notation instead of %ENV% to expand the environment variables.