Making HAProxy highly available with Keepalived
To remove HAProxy as a single point of failure, run HAProxy on two gateway hosts and use Keepalived to move a virtual IP (VIP) between them. Clients keep using the same load balancer URL when the VIP moves.
Keepalived implements the Virtual Router Redundancy Protocol (VRRP). The gateway with the higher priority holds the VIP. A tracking script checks whether HAProxy is running; if HAProxy stops, Keepalived lowers the priority of that gateway and the VIP moves to the other gateway, typically within 3 to 15 seconds.
- Two gateway hosts on the same network segment, each running HAProxy with the same haproxy.cfg file.
- An unused IP address on that segment for the VIP, and a VRRP router ID (1 to 255) that no other VRRP group on the segment uses.
- A DNS A record that resolves the load balancer FQDN, for example
livy-lb.example.com, to the VIP. Use this FQDN in the Livy Load Balancer Address. - Network rules that allow VRRP traffic (IP protocol 112) between the two gateway hosts.
Use the following table to troubleshoot common problems:
| Symptom | Action |
|---|---|
| The VIP is present on both gateway hosts. | Check that the priorities differ (101 and 100), that VRRP traffic can pass
between the hosts, and that no other VRRP group on the network segment uses the same
virtual_router_id. |
Requests through the VIP fail with HTTP 401 after a
failover. |
Verify that LIVY_LOAD_BALANCER_URL is set in every Livy Server
process, and restart the roles where it is missing. |
Requests through the VIP fail with HTTP 403. |
Verify that the keytab of every Livy Server role contains the
HTTP/load-balancer-fqdn principal. |
| Listing sessions through the VIP does not return all sessions. | This is expected. Each request reaches one backend, and each backend lists only its own sessions. |
To review recent events, run journalctl -u keepalived -n 50 --no-pager
and journalctl -u haproxy -n 50 --no-pager.
