Livy active/passive high availability

Livy high availability is active/passive: one Livy server is active and handles sessions; additional servers are passive and redirect requests to the active server with HTTP 307. When more than one Livy Server role is deployed in a cluster, Cloudera Manager enables HA by setting livy.server.recovery.mode to ha and configuring ZooKeeper.

In active/passive high availability mode, multiple Livy instances run in a cluster. Both active and passive instances run continuously; passive instances take the active role when required. There is only one active server at a time, and the remaining instances are passive. Passive instances redirect requests to the active server using HTTP 307.

Limitations

JDBC connection: The JDBC connections are not redirected. More details about this limitation are available below.
Load balancing: The active/passive high availability model does not provide additional parallelism or capacity. Interactive sessions are only handled by the active server; adding more servers does not distribute load across servers.

Using Livy without Knox gateway

If you are not using Livy through Knox gateway, clients must follow HTTP redirects and resend authentication. The logic the clients can use is to make a list of the Livy servers by obtaining them from the Cloudera Manager API and if any of the servers do not respond, the clients should retry sending the request to another server in the list. For example, in the case of cURL, the --location-trusted flag has to be specified to follow redirects and resend authentication.

Active/passive high availability and JDBC connections

Livy provides high availability through an active/passive setup. Only the active Livy server node can accept JDBC connections, and the passive nodes reject connection attempts. Therefore, if a client wants to connect using JDBC, it has to iterate through all servers and check which one accepts connections. If the active server goes down, the connection is broken and another node takes over the active role. This behavior is the same for both HTTP and binary mode connections.

Multiple independent Livy services

When you run more than one Livy service in a cluster (for example, Livy-for-Spark3-1 and Livy-for-Spark3-2), each service must use unique values for livy.server.port, livy.server.recovery.state-store.url, and livy.server.zk.key-prefix. Without unique HDFS recovery paths and ZooKeeper key prefixes, session recovery and leader election can conflict between services.

For step-by-step instructions, see Adding independent Livy services with active/passive HA.