Accessing the Spark History Server

Configure HDFS permissions on the Spark event log directory, then access the Spark History Server for your Spark cluster from the Cloudera Management Console interface.

The Spark History Server reads Spark event logs from HDFS. By default, logs are stored at /user/spark/applicationHistory. Spark 3 clusters may use /user/spark/spark3ApplicationHistory depending on your configuration. The spark user must have permission to read the event log directory before the History Server UI can list applications.

  1. Grant the spark user read, write, and execute permissions on /user/spark and its contents:
    hdfs dfs -setfacl -R -m user:spark:rwx /user/spark
    hdfs dfs -setfacl -R -m default:user:spark:rwx /user/spark

    Default ACLs ensure that all newly created files and subdirectories under /user/spark automatically inherit the same ACLs.

  2. Restart the Spark History Server service.
  3. In the Cloudera Management Console, navigate to your Spark cluster: Data Hub Clusters > Cluster Name>
  4. Select the Gateway tab.
  5. Click the URL for Spark History Server.