Using the Livy API to run Spark jobs
Using the Livy API to run Spark jobs is similar to using the original Spark API. Livy provides programmatic and REST APIs and supports Scala 2.12 for Spark 3 jobs.
Programmatic API
Livy provides a programmatic Java/Scala and Python API that allows applications to run code inside Spark without having to maintain a local Spark context. For more information, see Using the Programmatic API documentation.
REST API
Architecturally, the client creates a remote Spark cluster, initializes it, and submits jobs through REST APIs. The Livy server unwraps and rewraps the job, and then sends it to the remote SparkContext through RPC. While the job runs the client waits for the result, using the same path. The following diagram illustrates the process:
Scala support
Livy supports Scala version 2.12 with Spark 3 (Scala 2.12 build).
For default Scala builds, Livy automatically detects the correct Scala version and associated jar files.
If you require a different Spark-Scala combination, set
livy.spark.scalaVersion to the desired version so that Livy
uses the right jar files.
