Preparing Livy service instances for HAProxy load balancing

Each Livy for Spark 3 service instance behind HAProxy needs its own unique settings and the same Livy Load Balancer Address, so that every instance accepts requests and Kerberos authentication for the load balancer hostname.

  • At least two Livy for Spark 3 service instances, each with unique values for livy.server.zk.key-prefix, livy.server.recovery.state-store.url, and, if roles of different instances share a host, livy.server.port. For instructions, see Adding independent Livy services with active/passive HA.
  • Two or more Livy Server roles in each service instance, so that each instance has a passive role that can take over as leader.
  • A Cloudera Manager release that provides the Livy Load Balancer Address property for Livy for Spark 3.
  • The client-facing URL of the load balancer, for example https://livy-lb.example.com:8888. If you use Keepalived, this is the fully qualified domain name (FQDN) that resolves to the virtual IP.
Set the load balancer address
  1. In Cloudera Manager, select the first Livy for Spark 3 service instance and click Configuration.
  2. Search for Livy Load Balancer Address and enter the client-facing URL of the load balancer.

    Enter exactly one URI; a comma-separated list is not supported. The value must match the scheme, host, and port that HAProxy exposes. The property corresponds to the Livy configuration key livy.lb.uri.

  3. Click Save Changes.
  4. Repeat these steps for every other Livy for Spark 3 service instance behind HAProxy, using the same value.
Update Kerberos credentials
  1. If the cluster uses Kerberos, stop all Livy Server roles of every service instance behind HAProxy.
  2. Generate the missing Kerberos credentials in Cloudera Manager, or regenerate the keytabs of the Livy Server roles.

    When the Livy Load Balancer Address is set, Cloudera Manager adds the HTTP/load-balancer-fqdn@REALM principal to the Livy keytabs. Do not create or merge keytabs manually with kadmin or ktutil.

Restart and verify
  1. Start or restart all Livy Server roles. If other services were stopped, start ZooKeeper, HDFS, and YARN before the Livy Server roles.

    Until the roles restart, Cloudera Manager shows the Livy services with a stale configuration. For more information, see Stale Configurations.

  2. On each Livy Server host, verify that the Livy process received the load balancer address:
    pid=$(pgrep -f 'org.apache.livy.server.LivyServer' | head -1)
    tr '\0' '\n' < /proc/$pid/environ | grep LIVY_LOAD_BALANCER_URL

    The output must show the same URL on every host, for example LIVY_LOAD_BALANCER_URL=https://livy-lb.example.com:8888. The process environment is authoritative; the property can be absent from the generated livy.conf file.

  3. If the cluster uses Kerberos, verify on each Livy Server host that the keytab contains the load balancer principal:
    pid=$(pgrep -f 'org.apache.livy.server.LivyServer' | head -1)
    proc_dir=$(readlink -f /proc/$pid/cwd)
    klist -kt "$proc_dir/livy_for_spark3.keytab" | grep -i HTTP/

    The output must include HTTP/load-balancer-fqdn@REALM.

Every Livy for Spark 3 service instance accepts requests that clients send to the load balancer URL. Next, enable the /ha-status endpoint so that HAProxy can detect the leader of each instance.