Livy load balancing with HAProxy
You can spread Livy sessions across several Livy for Spark 3 service instances by placing HAProxy in front of them. Each service instance keeps its own active/passive high availability, and HAProxy distributes new sessions between the instances and keeps each existing session on the instance that created it.
How this differs from Livy active/passive high availability
In Livy active/passive high availability, ZooKeeper elects one leader per Livy service. Only the leader runs sessions; the other Livy Server roles of the same service are passive and redirect requests to the leader. Adding roles to a service improves availability, but does not add capacity.
The HAProxy pattern adds capacity by running several independent Livy for Spark 3 service instances, each with its own leader. HAProxy treats each service instance as one backend and sends traffic only to the current leader of that backend. Active/passive high availability still protects each backend.
Architecture
- Clients use a single URL, for example
https://livy-lb.example.com:8888. The same URL is set as the Livy Load Balancer Address in Cloudera Manager. - Backends: each Livy for Spark 3 service instance in Cloudera Manager is one HAProxy backend. Every server line in a backend is one Livy Server role of that service instance.
- Health checks: HAProxy polls the
/ha-statusendpoint of each Livy server. Only the leader returnsHTTP 200, so HAProxy sends requests directly to the leader instead of relying onHTTP 307redirects. - Gateway redundancy (optional): two gateway hosts run HAProxy with identical configuration, and Keepalived moves a virtual IP (VIP) between them, so HAProxy is not a single point of failure.
Sticky sessions
Livy session IDs are local to one service instance; backends do not share sessions. HAProxy therefore keeps each client on one backend with a cookie:
- A request without a cookie, such as the request that creates a new session, goes to the default backend, which contains every Livy server.
- HAProxy forwards the request to a leader and returns a
SRV_IDcookie that identifies the server. - The client sends the cookie with every later request. HAProxy maps the cookie value to the backend that owns the session.
The client must store and resend the cookie, for example with a cookie jar. A client that reuses its cookie stays on the same backend; to place sessions on other backends, the client creates new sessions without the cookie. Listing sessions through HAProxy returns only the sessions of the backend that receives the request.
Failover behavior
- Livy leader failure: ZooKeeper elects a new leader within the same service instance, and the next health check marks it as available. Sessions recover from the instance's own recovery state store.
- Whole service instance unavailable: HAProxy stops sending new sessions to that backend. Existing sessions of that instance are unavailable until the instance recovers; they do not move to another backend.
- Gateway failure: with Keepalived, the VIP moves to the other gateway within seconds. Requests that are in flight during the switch can fail and must be retried.
Limitations with multiple Livy for Spark 3 services
Because this pattern adds Livy for Spark 3 services to the cluster, be aware of the following limitations:
- Gateway role: Cloudera Manager supports the Livy gateway role only on the first Livy for Spark 3 service. You cannot add the gateway role to the second or any later Livy for Spark 3 service. Add the gateway role to the first service only, and point clients to the load balancer URL instead of to an individual service instance.
- Knox topologies: Knox service discovery adds the Livy
Server roles of every Livy for Spark 3 service to the
cdp-proxy,cdp-proxy-api, andcdp-proxy-tokentopologies. To expose only the Livy servers that you intend, set the Livy URLs explicitly, as described in Configuring Knox topologies for multiple Livy for Spark 3 services.
