6. Smoke Test MapReduce

  1. Browse to the ResourceManager:

    http://$resourcemanager.full.hostname:8088/

  2. Create a $CLIENT_USER in all of the nodes and add it to the users group.

    useradd client
    usermod -a -G users client
  3. As the HDFS user, create a /user/$CLIENT_USER.

    sudo su $HDFS_USER 
    hdfs dfs -mkdir /user/$CLIENT_USER
    hdfs dfs -chown $CLIENT_USER:$CLIENT_USER /user/$CLIENT_USER 
    hdfs dfs -chmod -R 755 /user/$CLIENT_USER
  4. Run the smoke test as the $CLIENT_USER. Using Terasort, sort 10GB of data.

    su $CLIENT_USER
    /usr/hdp/current/hadoop-client/bin/hadoop jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples-*.jar teragen 10000 tmp/teragenout
    /usr/hdp/current/hadoop-client/bin/hadoop jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples-*.jar terasort tmp/teragenout tmp/terasortout

loading table of contents...