Configuring HAProxy for Livy

Configure HAProxy with one backend per Livy for Spark 3 service instance, cookie-based sticky sessions, and /ha-status health checks, and then verify that sessions stay on the backend that created them.

  • The Livy for Spark 3 service instances are prepared and the /ha-status endpoint is enabled on every Livy server.
  • One or two gateway hosts outside the Livy roles, with HAProxy installed from the operating system packages. HAProxy is not managed by Cloudera Manager.
  • A TLS certificate and private key for the load balancer FQDN in one PEM file, and the CA certificate that signed the Livy server certificates.
  1. On the gateway host, edit /etc/haproxy/haproxy.cfg. The following example configures two Livy for Spark 3 service instances with two Livy Server roles each:
    global
        log         127.0.0.1 local2
        chroot      /var/lib/haproxy
        maxconn     4000
        user        haproxy
        group       haproxy
        daemon
    
    defaults
        mode                    http
        log                     global
        option                  httplog
        option                  http-server-close
        option                  forwardfor       except 127.0.0.0/8
        option                  redispatch
        timeout http-request    10s
        timeout connect         10s
        timeout client          1m
        timeout server          1m
    
    frontend livy_gateway
        bind *:8888 ssl crt /path/to/gateway.pem
    
        acl cookie_livy1 cook(SRV_ID) -m str s1 s2
        use_backend livy1 if cookie_livy1
    
        acl cookie_livy2 cook(SRV_ID) -m str s3 s4
        use_backend livy2 if cookie_livy2
    
        default_backend livy_default
    
    backend livy_default
        balance roundrobin
        cookie SRV_ID insert indirect nocache
        option httpchk GET /ha-status
        http-check expect status 200
        option log-health-checks
        retries 3
        server livy1-a livy1-a.example.com:28998 check inter 5s fall 3 rise 2 cookie s1 ssl ca-file /path/to/ca.pem
        server livy1-b livy1-b.example.com:28998 check inter 5s fall 3 rise 2 cookie s2 ssl ca-file /path/to/ca.pem
        server livy2-a livy2-a.example.com:28998 check inter 5s fall 3 rise 2 cookie s3 ssl ca-file /path/to/ca.pem
        server livy2-b livy2-b.example.com:28998 check inter 5s fall 3 rise 2 cookie s4 ssl ca-file /path/to/ca.pem
    
    backend livy1
        balance roundrobin
        cookie SRV_ID insert indirect nocache
        option httpchk GET /ha-status
        http-check expect status 200
        option log-health-checks
        retries 3
        server livy1-a livy1-a.example.com:28998 check inter 5s fall 3 rise 2 cookie s1 ssl ca-file /path/to/ca.pem
        server livy1-b livy1-b.example.com:28998 check inter 5s fall 3 rise 2 cookie s2 ssl ca-file /path/to/ca.pem
    
    backend livy2
        balance roundrobin
        cookie SRV_ID insert indirect nocache
        option httpchk GET /ha-status
        http-check expect status 200
        option log-health-checks
        retries 3
        server livy2-a livy2-a.example.com:28998 check inter 5s fall 3 rise 2 cookie s3 ssl ca-file /path/to/ca.pem
        server livy2-b livy2-b.example.com:28998 check inter 5s fall 3 rise 2 cookie s4 ssl ca-file /path/to/ca.pem

    Replace the host names, ports, and file paths with the values for your cluster. Each Livy Server role needs one server line with a unique cookie value, and the cookie values in each acl line must match the server lines of the corresponding backend. If roles of different service instances share a host, use the livy.server.port value of each instance.

    The key directives work as follows:

    Directive Purpose
    bind *:8888 ssl crt Accepts client connections over TLS on the port of the Livy Load Balancer Address.
    acl cookie_livy1 cook(SRV_ID) -m str s1 s2 and use_backend livy1 if cookie_livy1 Sends a client whose SRV_ID cookie is s1 or s2 to the livy1 backend. HAProxy evaluates these rules first, so returning clients always follow their cookie.
    default_backend livy_default Sends clients without a matching cookie, such as clients that create a new session, to a backend that contains every Livy server.
    cookie SRV_ID insert indirect nocache Adds the SRV_ID cookie to responses so that the client stays on the same server.
    option httpchk GET /ha-status and http-check expect status 200 Marks only the leader of each service instance as available.
    check inter 5s fall 3 rise 2 Runs the health check every 5 seconds, marks a server unavailable after 3 failed checks, and available again after 2 successful checks.

    For connection limits and timeouts for many concurrent sessions, see Livy and HAProxy tuning reference.

  2. Check the configuration file for errors:
    haproxy -c -f /etc/haproxy/haproxy.cfg
  3. Enable and start HAProxy:
    systemctl enable --now haproxy

    After later changes to the configuration file, apply them with systemctl reload haproxy.

  4. If you use two gateway hosts, copy the same haproxy.cfg file to the second gateway host and repeat the previous steps there.
  5. Verify sticky sessions through the load balancer. Obtain a Kerberos ticket for a user that is allowed to use Livy, and then create a session with a cookie file:
    curl --location --location-trusted -k -X POST --negotiate -u : \
      -c cookies.dat -b cookies.dat \
      -H "Content-Type: application/json" \
      -d '{"kind": "spark"}' \
      https://livy-lb.example.com:8888/sessions

    Use the same cookie file for every request in the test.

  6. Submit a statement and poll for its result, replacing session-id with the ID from the previous response:
    curl --location --location-trusted -k -X POST --negotiate -u : \
      -c cookies.dat -b cookies.dat \
      -H "Content-Type: application/json" \
      -d '{"code": "2+3"}' \
      https://livy-lb.example.com:8888/sessions/session-id/statements
    
    curl --location --location-trusted -k --negotiate -u : \
      -c cookies.dat -b cookies.dat \
      https://livy-lb.example.com:8888/sessions/session-id/statements/0
  7. Check the cookie file.

    The file contains an SRV_ID entry for the load balancer host, for example s1. This value identifies the server line, and therefore the backend, that owns the session.

The session is created, the statement reaches the available state, and all requests for the session use the same SRV_ID cookie. To place a session on another backend, create it with a new cookie file.