Command Line Installation
Also available as:
PDF
loading table of contents...

Configuring Livy

Perform the following steps to configure Livy:

  1. Login as root, or use root privilege to create users Livy and Zeppelin:

    useradd livy -g hadoop
    useradd zeppelin -g hadoop
  2. Create a log directory for Livy:

    mkdir /var/log/livy
  3. Change owner of /var/log/livy to livy:hadoop.

    chown livy:hadoop /var/log/livy
  4. /etc/livy/livy.conf contains information regarding server configuration.

    Create file /etc/livy/livy.conf and add the following to the file:

    livy.environment production
    livy.impersonation.enabled true
    livy.server.csrf_protection.enabled true
    livy.server.port 8998
    livy.server.session.timeout 3600000
  5. /etc/livy/spark-blacklist.conf defines a list of properties that users are not allowed to override when a Spark session is started.

    Create an empty file called /etc/livy/spark-blacklist.conf.

  6. Create file /etc/livy/livy-env.sh to define the environmental variables. Add the following to the file:

    export SPARK_HOME=/usr/hdp/current/spark-client
    export LIVY_LOG_DIR=/var/log/livy
    export LIVY_PID_DIR=/var/run/livy
    export LIVY_SERVER_JAVA_OPTS="-Xmx2g"
  7. If you are not using Kerberos, skip these steps.

    1. If you are using Kerberos, create Livy and Zeppelin principals and keytabs:

      kadmin.local -q "addprinc -randkey livy@EXAMPLE.COM"
      kadmin.local -q "xst -k /etc/security/keytabs/livy.headless.keytab livy@EXAMPLE.COM"
      kadmin.local -q "addprinc -randkey zeppelin@EXAMPLE.COM"
      kadmin.local -q "xst -k /etc/security/keytabs/zeppelin.headless.keytab zeppelin@EXAMPLE.COM"
      
    2. If you are using Kerberos, move the Livy and Zeppelin key tables to the node on which Livy and Zeppelin will run.

      chown livy:hadoop /etc/security/keytabs/livy.headless.keytab
      chown zeppelin:hadoop /etc/security/keytabs/zeppelin.headless.keytab
      
  8. Ensure that the Livy user can read the contents of the /etc/livy/conf directory.