In HDP 2.6, Livy supports both Spark1 and Spark2 in one build. To specify which version of Spark to use, set SPARK_HOME to Spark1 or Spark2. Livy automatically differentiates between the two.

Here is a sample export statement in the livy-env.sh file:

export SPARK_HOME=<path-to>/spark-2.1.0-bin-hadoop2.6

Scala Support

Livy supports Scala versions 2.10 and 2.11.

For default Scala builds, Spark 1.6 with Scala 2.10 or Spark 2.0 with Scala 2.11, Livy automatically detects the correct Scala version and associated jar files.

If you require a different Spark-Scala combination, such as Spark 2.0 with Scala 2.10, set livy.spark.scalaVersion to the desired version so that Livy uses the right jar files.