Chapter 7. Troubleshooting Spark
When you run a Spark job, you will see a standard set of console messages. In addition, the following information is available:
A list of running applications, where you can retrieve the application ID and check the application log:
yarn application –listyarn logs -applicationId <app_id>Check the Spark environment for a specific job:
http://<host>:8088/proxy/<job_id>/environment/
Specific Issues
The following paragraphs describe specific issues and possible solutions:
Issue: Job stays in "accepted" state; it doesn't run. This can happen when a job requests more memory or cores than available.
Solution: Assess workload to see if any resources can be released. You might need to stop unresponsive jobs to make room for the job.
Issue: Insufficient HDFS access. This can lead to errors such as the following:
“Loading data to table default.testtable Failed with exception Unable to move sourcehdfs://blue1:8020/tmp/hive-spark/hive_2015-03-04_ 12-45-42_404_3643812080461575333-1/-ext-10000/kv1.txt to destination hdfs://blue1:8020/apps/hive/warehouse/testtable/kv1.txt”
Solution: Make sure the user or group running the job has sufficient HDFS privileges to the location.
Issue: Wrong host in Beeline, shows error as invalid URL:
Error: Invalid URL: jdbc:hive2://localhost:10001 (state=08S01,code=0)
Solution: Specify the correct Beeline host assignment.
Issue: Error: closed SQLContext.
Solution: Restart the Thrift server.