Tuning Apache Spark
Also available as:
PDF

Check Job History

You can use the following resources to view job history:

  • Spark history server UI: view information about Spark jobs that have completed.

    1. On an Ambari-managed cluster, in the Ambari Services tab, select Spark.
    2. Click Quick Links.
    3. Choose the Spark history server UI.

      Ambari displays a list of jobs.

    4. Click "App ID" for job details.
  • Spark history server web UI: view information about Spark jobs that have completed.

    In a browser window, navigate to the history server web UI. The default host port is<host>:18080.

  • YARN web UI: view job history and time spent in various stages of the job:

    http://<host>:8088/proxy/<job_id>/environment/

    http://<host>:8088/proxy/<app_id>/stages/

  • yarn logs command: list the contents of all log files from all containers associated with the specified application.

    yarn logs -applicationId <app_id>.

  • Hadoop Distributed File System (HDFS) shell or API: view container log files.