Making HAProxy highly available with Keepalived

To remove HAProxy as a single point of failure, run HAProxy on two gateway hosts and use Keepalived to move a virtual IP (VIP) between them. Clients keep using the same load balancer URL when the VIP moves.

Keepalived implements the Virtual Router Redundancy Protocol (VRRP). The gateway with the higher priority holds the VIP. A tracking script checks whether HAProxy is running; if HAProxy stops, Keepalived lowers the priority of that gateway and the VIP moves to the other gateway, typically within 3 to 15 seconds.

  • Two gateway hosts on the same network segment, each running HAProxy with the same haproxy.cfg file.
  • An unused IP address on that segment for the VIP, and a VRRP router ID (1 to 255) that no other VRRP group on the segment uses.
  • A DNS A record that resolves the load balancer FQDN, for example livy-lb.example.com, to the VIP. Use this FQDN in the Livy Load Balancer Address.
  • Network rules that allow VRRP traffic (IP protocol 112) between the two gateway hosts.
  1. On both gateway hosts, install and enable Keepalived from the operating system packages, for example:
    yum install -y keepalived
    systemctl enable keepalived
  2. On the first gateway host, create /etc/keepalived/keepalived.conf with the following content:
    global_defs {
        script_user root
        enable_script_security
    }
    
    vrrp_script chk_haproxy {
        script "/usr/bin/systemctl is-active haproxy"
        interval 2
        weight -30
        fall 2
        rise 2
    }
    
    vrrp_instance VI_LIVY {
        state MASTER
        interface nic
        virtual_router_id router-id
        priority 101
        advert_int 1
        preempt_delay 5
    
        authentication {
            auth_type PASS
            auth_pass vrrp-password
        }
    
        virtual_ipaddress {
            vip/prefix-length dev nic
        }
    
        track_script {
            chk_haproxy
        }
    }

    Replace nic with the network interface that carries the VIP, router-id with the VRRP router ID, vip and prefix-length with the VIP and its network prefix length, and vrrp-password with a shared password.

    The weight -30 setting lowers the priority of this gateway from 101 to 71 when HAProxy is not running, which is below the priority of the second gateway.

  3. On the second gateway host, create the same file with two changes: set state BACKUP and priority 100.

    All other values, including virtual_router_id, auth_pass, and the VIP, must be identical on both hosts.

  4. Start Keepalived on both gateway hosts:
    systemctl restart keepalived
    systemctl is-active keepalived
  5. Verify that only the first gateway host holds the VIP. Run the following command on each gateway host:
    ip -4 addr show nic | grep vip

    The VIP appears on the first gateway host and not on the second.

  6. Optional: To monitor HAProxy through the VIP, add a statistics listener to haproxy.cfg on both gateway hosts and reload HAProxy:
    listen admin
        bind *:8000
        stats enable
        stats uri /stats

    You can then check the active gateway with curl http://vip:8000/stats;csv.

  7. Test failover. On the first gateway host, stop HAProxy:
    systemctl stop haproxy

    Within a few seconds, the VIP moves to the second gateway host. While the VIP moves, requests through the load balancer can fail, for example with HTTP 504. This is expected. Do not create sessions during the transition; retry requests when the load balancer responds again.

  8. Start HAProxy again on the first gateway host:
    systemctl start haproxy

    After the preemption delay, the VIP returns to the first gateway host. Existing sessions and SRV_ID cookies remain valid, because both gateways use the same HAProxy configuration.

Use the following table to troubleshoot common problems:

Symptom Action
The VIP is present on both gateway hosts. Check that the priorities differ (101 and 100), that VRRP traffic can pass between the hosts, and that no other VRRP group on the network segment uses the same virtual_router_id.
Requests through the VIP fail with HTTP 401 after a failover. Verify that LIVY_LOAD_BALANCER_URL is set in every Livy Server process, and restart the roles where it is missing.
Requests through the VIP fail with HTTP 403. Verify that the keytab of every Livy Server role contains the HTTP/load-balancer-fqdn principal.
Listing sessions through the VIP does not return all sessions. This is expected. Each request reaches one backend, and each backend lists only its own sessions.

To review recent events, run journalctl -u keepalived -n 50 --no-pager and journalctl -u haproxy -n 50 --no-pager.