Setting Python path variables for Livy
To change the Python executable used by a Livy session, follow the instructions for your version of Python.
pyspark
Livy reads the path from the PYSPARK_PYTHON
environment variable (this
is the same as PySpark).
-
If Livy is running in
local
mode, simply set the environment variable (this is the same as PySpark). -
If the Livy session is running in
yarn-cluster
mode, setspark.yarn.appMasterEnv.PYSPARK_PYTHON
in theSparkConf
file, so that the environment variable is passed to the driver.
pyspark3
Livy reads the path from environment variable PYSPARK3_PYTHON
.
-
If Livy is running in
local
mode, simply set the environment variable. -
If the Livy session is running in
yarn-cluster
mode, setspark.yarn.appMasterEnv.PYSPARK3_PYTHON
inSparkConf
file, so that the environment variable is passed to the driver.