Upgrading HDP Manually
Also available as:
PDF
loading table of contents...

Run Hadoop Smoke Tests

To smoke test your Hadoop upgrade, you can run the following MapReduce job as a regular user.

The job uses MapReduce to write 100MB of data into HDFS with RandomWriter

hadoop jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar 
            
randomwriter -Dtest.randomwrite.total_bytes=10000000 test-after-upgrade.

You should see messages similar to:

map 0% reduce 0%
…map 100% reduce 100%
Job …. completed successfully

MapReduce upgraded successfully. You can now upgrade your other components.

Basic Troubleshooting

To find the number of active nodes and NodeManagers, access the ResourceManager web UI:

http://<resource manager host>:8088/cluster/nodes

The number of active nodes should be equal to the number of nodemanagers.

Accessing error messages:

  1. Access the ApplicationMaster WebUI to view the container logs.

  2. At your console logs for MapReduce job, look for a line with this format:

    13/10/02 17:57:21 INFO mapreduce.Job: The url to track the job: http://<resource manager host>:8088/proxy/application_1380673658357_0007/

  3. Select the logs link under ApplicationMaster table. It will redirect you to the container logs. Error messages display here.