Configuring HAProxy for Livy
Configure HAProxy with one backend per Livy for Spark 3 service instance, cookie-based
sticky sessions, and /ha-status health checks, and then verify that sessions
stay on the backend that created them.
- The Livy for Spark 3 service instances are prepared and the
/ha-statusendpoint is enabled on every Livy server. - One or two gateway hosts outside the Livy roles, with HAProxy installed from the operating system packages. HAProxy is not managed by Cloudera Manager.
- A TLS certificate and private key for the load balancer FQDN in one PEM file, and the CA certificate that signed the Livy server certificates.
-
On the gateway host, edit /etc/haproxy/haproxy.cfg. The following
example configures two Livy for Spark 3 service instances with two Livy Server roles
each:
global log 127.0.0.1 local2 chroot /var/lib/haproxy maxconn 4000 user haproxy group haproxy daemon defaults mode http log global option httplog option http-server-close option forwardfor except 127.0.0.0/8 option redispatch timeout http-request 10s timeout connect 10s timeout client 1m timeout server 1m frontend livy_gateway bind *:8888 ssl crt /path/to/gateway.pem acl cookie_livy1 cook(SRV_ID) -m str s1 s2 use_backend livy1 if cookie_livy1 acl cookie_livy2 cook(SRV_ID) -m str s3 s4 use_backend livy2 if cookie_livy2 default_backend livy_default backend livy_default balance roundrobin cookie SRV_ID insert indirect nocache option httpchk GET /ha-status http-check expect status 200 option log-health-checks retries 3 server livy1-a livy1-a.example.com:28998 check inter 5s fall 3 rise 2 cookie s1 ssl ca-file /path/to/ca.pem server livy1-b livy1-b.example.com:28998 check inter 5s fall 3 rise 2 cookie s2 ssl ca-file /path/to/ca.pem server livy2-a livy2-a.example.com:28998 check inter 5s fall 3 rise 2 cookie s3 ssl ca-file /path/to/ca.pem server livy2-b livy2-b.example.com:28998 check inter 5s fall 3 rise 2 cookie s4 ssl ca-file /path/to/ca.pem backend livy1 balance roundrobin cookie SRV_ID insert indirect nocache option httpchk GET /ha-status http-check expect status 200 option log-health-checks retries 3 server livy1-a livy1-a.example.com:28998 check inter 5s fall 3 rise 2 cookie s1 ssl ca-file /path/to/ca.pem server livy1-b livy1-b.example.com:28998 check inter 5s fall 3 rise 2 cookie s2 ssl ca-file /path/to/ca.pem backend livy2 balance roundrobin cookie SRV_ID insert indirect nocache option httpchk GET /ha-status http-check expect status 200 option log-health-checks retries 3 server livy2-a livy2-a.example.com:28998 check inter 5s fall 3 rise 2 cookie s3 ssl ca-file /path/to/ca.pem server livy2-b livy2-b.example.com:28998 check inter 5s fall 3 rise 2 cookie s4 ssl ca-file /path/to/ca.pemReplace the host names, ports, and file paths with the values for your cluster. Each Livy Server role needs one
serverline with a unique cookie value, and the cookie values in eachaclline must match the server lines of the corresponding backend. If roles of different service instances share a host, use thelivy.server.portvalue of each instance.The key directives work as follows:
Directive Purpose bind *:8888 ssl crtAccepts client connections over TLS on the port of the Livy Load Balancer Address. acl cookie_livy1 cook(SRV_ID) -m str s1 s2anduse_backend livy1 if cookie_livy1Sends a client whose SRV_IDcookie iss1ors2to thelivy1backend. HAProxy evaluates these rules first, so returning clients always follow their cookie.default_backend livy_defaultSends clients without a matching cookie, such as clients that create a new session, to a backend that contains every Livy server. cookie SRV_ID insert indirect nocacheAdds the SRV_IDcookie to responses so that the client stays on the same server.option httpchk GET /ha-statusandhttp-check expect status 200Marks only the leader of each service instance as available. check inter 5s fall 3 rise 2Runs the health check every 5 seconds, marks a server unavailable after 3 failed checks, and available again after 2 successful checks. For connection limits and timeouts for many concurrent sessions, see Livy and HAProxy tuning reference.
-
Check the configuration file for errors:
haproxy -c -f /etc/haproxy/haproxy.cfg -
Enable and start HAProxy:
systemctl enable --now haproxyAfter later changes to the configuration file, apply them with systemctl reload haproxy.
- If you use two gateway hosts, copy the same haproxy.cfg file to the second gateway host and repeat the previous steps there.
-
Verify sticky sessions through the load balancer. Obtain a Kerberos ticket for a user
that is allowed to use Livy, and then create a session with a cookie file:
curl --location --location-trusted -k -X POST --negotiate -u : \ -c cookies.dat -b cookies.dat \ -H "Content-Type: application/json" \ -d '{"kind": "spark"}' \ https://livy-lb.example.com:8888/sessionsUse the same cookie file for every request in the test.
-
Submit a statement and poll for its result, replacing session-id
with the ID from the previous response:
curl --location --location-trusted -k -X POST --negotiate -u : \ -c cookies.dat -b cookies.dat \ -H "Content-Type: application/json" \ -d '{"code": "2+3"}' \ https://livy-lb.example.com:8888/sessions/session-id/statements curl --location --location-trusted -k --negotiate -u : \ -c cookies.dat -b cookies.dat \ https://livy-lb.example.com:8888/sessions/session-id/statements/0 -
Check the cookie file.
The file contains an
SRV_IDentry for the load balancer host, for examples1. This value identifies the server line, and therefore the backend, that owns the session.
available state, and all requests for the session use the same
SRV_ID cookie. To place a session on another backend, create it with a new
cookie file.