Check Job History

You can use the following resources to view job history:

• Spark history server UI: view information about Spark jobs that have completed.

  1. On an Ambari-managed cluster, in the Ambari Services tab, select Spark.
  2. Click Quick Links.
  3. Choose the Spark history server UI.

     Ambari displays a list of jobs.

  4. Click "App ID" for job details.

• Spark history server web UI: view information about Spark jobs that have completed.

  In a browser window, navigate to the history server web UI. The default host port is<host>:18080.

• YARN web UI: view job history and time spent in various stages of the job:

  http://<host>:8088/proxy/<job_id>/environment/

  http://<host>:8088/proxy/<app_id>/stages/

• yarn logs command: list the contents of all log files from all containers associated with the specified application.

  yarn logs -applicationId <app_id>.

• Hadoop Distributed File System (HDFS) shell or API: view container log files.