Preparing Livy service instances for HAProxy load balancing
Each Livy for Spark 3 service instance behind HAProxy needs its own unique settings and
the same Livy Load Balancer Address, so that every instance accepts requests and Kerberos
authentication for the load balancer hostname.
At least two Livy for Spark 3 service instances, each with unique values for
livy.server.zk.key-prefix,
livy.server.recovery.state-store.url, and, if roles of different
instances share a host, livy.server.port. For instructions, see Adding independent Livy services
with active/passive HA.
Two or more Livy Server roles in each service instance, so that each instance has a
passive role that can take over as leader.
A Cloudera Manager release that provides the Livy Load
Balancer Address property for Livy for Spark 3.
The client-facing URL of the load balancer, for example
https://livy-lb.example.com:8888. If you use Keepalived, this is the
fully qualified domain name (FQDN) that resolves to the virtual IP.
Set the load balancer address
In Cloudera Manager, select the first Livy for Spark 3 service
instance and click Configuration.
Search for Livy Load Balancer Address and enter the
client-facing URL of the load balancer.
Enter exactly one URI; a comma-separated list is not supported. The value must match
the scheme, host, and port that HAProxy exposes. The property corresponds to the Livy
configuration key livy.lb.uri.
Click Save Changes.
Repeat these steps for every other Livy for Spark 3 service instance behind HAProxy,
using the same value.
Update Kerberos credentials
If the cluster uses Kerberos, stop all Livy Server roles of every service instance
behind HAProxy.
Generate the missing Kerberos credentials in Cloudera Manager, or
regenerate the keytabs of the Livy Server roles.
When the Livy Load Balancer Address is set, Cloudera Manager adds the
HTTP/load-balancer-fqdn@REALM
principal to the Livy keytabs. Do not create or merge keytabs manually with
kadmin or ktutil.
Restart and verify
Start or restart all Livy Server roles. If other services were stopped, start
ZooKeeper, HDFS, and YARN before the Livy Server roles.
Until the roles restart, Cloudera Manager shows the Livy services
with a stale configuration. For more information, see Stale Configurations.
On each Livy Server host, verify that the Livy process received the load balancer
address:
The output must show the same URL on every host, for example
LIVY_LOAD_BALANCER_URL=https://livy-lb.example.com:8888. The process
environment is authoritative; the property can be absent from the generated
livy.conf file.
If the cluster uses Kerberos, verify on each Livy Server host that the keytab contains
the load balancer principal:
The output must include
HTTP/load-balancer-fqdn@REALM.
Every Livy for Spark 3 service instance accepts requests that
clients send to the load balancer URL. Next, enable the /ha-status endpoint
so that HAProxy can detect the leader of each instance.