Cloudera Docs

Adding Spark to ML Runtime Sessions

You can add Spark to ML Runtime Sessions using the ML Runtimes Addons. Both Spark and Hadoop CLI are enabled when you enable Spark.

To add Spark to sessions run on projects using ML Runtimes images:

  1. You can view all available ML Runtime addons by selecting the Site Administration>Runtime/Engine tab and viewing Runtime Addons.
  2. Start a New Session for an ML Runtimes project.
  3. Click the Enable Spark option, then select the Spark version.
  4. Click Start Session.
  5. You can now run a Spark job for your session.
    The Logs tab displays the executors in your Spark job.
  6. After you start a Spark job, you can access the Spark UI by clicking the Spark UI button at the top of the Session window.