Cloudera Docs

Setting Python path variables for Livy

Livy uses the Python executable and PYTHONPATH as set by the Cloudera Manager Agent for PySpark sessions. If required, it is possible to override these default settings via multiple environment variables.

You can override the default settings for PYTHONPATH using the following methods:

  1. In Cloudera Manager > Clusters > Livy > Configuration > Override CM PYTHONPATH enabled check the checkbox.
  2. In Cloudera Manager > Clusters > Livy > Configuration > Override CM PYTHONPATH value enter the new Livy Server Default Group.
  1. In Cloudera Manager > Clusters > LIVY_FOR_SPARK3 > Configuration > Override CM PYTHONPATH enabled check the Livy Server for Spark 3 Default Group checkbox.
  2. In Cloudera Manager > Clusters > LIVY_FOR_SPARK3 > Configuration > Override CM PYTHONPATH value enter the Livy Server for Spark 3 Default Group value.
  1. In Cloudera Manager > Clusters > SPARK3_ON_YARN > Configuration > Spark 3 Client Advanced Configuration Snippet (Safety Valve) for spark3-conf/spark-defaults.conf enter the following values:
    • spark.yarn.appMasterEnv.PYSPARK_DRIVER_PYTHON=/path/to/python_binary
    • spark.yarn.appMasterEnv.PYSPARK_PYTHON=/path/to/python_binary
    • spark.yarn.executorEnv.PYSPARK_PYTHON=/path/to/python_binary
  1. In Cloudera Manager > Clusters > SPARK_ON_YARN > Configuration > Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.conf enter the following values:
    • spark.yarn.appMasterEnv.PYSPARK_DRIVER_PYTHON=/path/to/python_binary
    • spark.yarn.appMasterEnv.PYSPARK_PYTHON=/path/to/python_binary
    • spark.yarn.executorEnv.PYSPARK_PYTHON=/path/to/python_binary