Cloudera Docs

Adding Spark to ML Runtime Sessions

You can add Spark to ML Runtime sessions using the ML Runtimes add-ons. Both Spark and Hadoop CLI are enabled when you enable Spark.

To add Spark to sessions run on projects using ML Runtimes images:

  1. View all available ML Runtime add-ons by selecting the Site Administration > Runtimes.
  2. Start a New Session for an ML Runtimes project.
  3. Click the Enable Spark option, then select the Spark version.
  4. Click Start Session.
  5. You can now run a Spark job for your session.
    The Logs tab displays the executors in your Spark job.
  6. After you start a Spark job, you can access the Spark UI by clicking the Spark UI button at the top of the Session window.