Spark 2 Web UIs (CDSW_SPARK_PORT)

Spark 2 exposes one web UI for each Spark application driver running in Cloudera Data Science Workbench. The UI will be running within the container, on the port specified by the environmental variable CDSW_SPARK_PORT.

By default, CDSW_SPARK_PORT is set to 20049. The web UI will exist only as long as a SparkContext is active within a session. The port is freed up when the SparkContext is shutdown.

Spark 2 web UI is available as a tab in the session, or alternatively in browsers at https://spark-<$CDSW_ENGINE_ID>".<$CDSW_DOMAIN>. For a running job, navigate to the Job Overview page and click the History tab. Click on the running job and select Spark UI.