Using Spark history server to troubleshoot Spark jobs

The Spark history server is a monitoring tool that displays information about completed Spark applications. It provides information for debugging such as Spark configurations, DAG execution, driver and executor resource utilization, application logs, and job, stage and task-level details.

To view spark history server information for a job run:

1. From the CDE Overview page, select the CDE service for the job you want to troubleshoot.
2. In the Virtual Clusters column, click View Jobs for the cluster containing the job.
3. Select the job you want to troubleshoot.
4. In the Jobs Run page, click the Job ID for the job run you want to view the information.
5. Click the Spark UI tab to access the Spark History Server.