Setting Python path variables for Livy

To change the Python executable used by a Livy session, set environment variables or Spark configuration when you create the session.

pyspark

Livy reads the path from the PYSPARK_PYTHON environment variable (this is the same as PySpark).

• If Livy is running in local mode, simply set the environment variable (this is the same as PySpark).

• If the Livy session is running in yarn-cluster mode, setspark.yarn.appMasterEnv.PYSPARK_PYTHON in the SparkConf file, so that the environment variable is passed to the driver.

• When you create a session through the Livy REST API, set spark.pyspark.python and spark.yarn.appMasterEnv.PYSPARK_PYTHON in the session conf map.