Livy active/passive high availability

Livy high availability is active/passive: one Livy server is active and handles sessions; additional servers are passive and redirect requests to the active server with HTTP 307. When more than one Livy Server role is deployed in a cluster and Cloudera Manager Enable active-passive HA mode (livy_ha_enabled_if_possible, default true) is enabled, Cloudera Manager sets livy.server.recovery.mode to ha and configures ZooKeeper.

In active/passive high availability mode, multiple Livy instances run in a cluster. Both active and passive instances run continuously; passive instances take the active role when required. There is only one active server at a time, and the remaining instances are passive. Passive instances redirect requests to the active server using HTTP 307.

Cloudera Manager control of HA

Cloudera Manager can enable active/passive HA automatically when your cluster runs more than one Livy Server role instance. The behavior is controlled by Enable active-passive HA mode (livy_ha_enabled_if_possible). The default value is true.

When livy_ha_enabled_if_possible is true and Cloudera Manager detects more than one Livy Server instance, Cloudera Manager sets livy.server.recovery.mode to ha, configures ZooKeeper for Livy, and writes the effective settings to livy.conf when you restart Livy or deploy client configuration. While this setting is enabled, values you add for livy.server.recovery.mode in the Livy Server for Spark 3 Advanced Configuration Snippet (Safety Valve) for livy3-conf/livy.conf do not take effect if they conflict with HA auto-enablement. The Cloudera Manager UI can show your Safety Valve edit, but the deployed livy.conf can still use ha.

To run Livy with livy.server.recovery.mode set to recovery or to manage recovery mode only through the Safety Valve, set livy_ha_enabled_if_possible to false, add livy.server.recovery.mode in the Safety Valve for livy3-conf/livy.conf, restart the Livy service, and deploy client configuration.

Limitations

JDBC connection: The JDBC connections are not redirected. More details about this limitation are available below.
Load balancing: The active/passive high availability model does not provide additional parallelism or capacity. Interactive sessions are only handled by the active server; adding more servers does not distribute load across servers.

Using Livy without Knox gateway

If you are not using Livy through Knox gateway, clients must follow HTTP redirects and resend authentication. The logic the clients can use is to make a list of the Livy servers by obtaining them from the Cloudera Manager API and if any of the servers do not respond, the clients should retry sending the request to another server in the list. For example, in the case of cURL, the --location-trusted flag has to be specified to follow redirects and resend authentication.

Active/passive high availability and JDBC connections

Livy provides high availability through an active/passive setup. Only the active Livy server node can accept JDBC connections, and the passive nodes reject connection attempts. Therefore, if a client wants to connect using JDBC, it has to iterate through all servers and check which one accepts connections. If the active server goes down, the connection is broken and another node takes over the active role. This behavior is the same for both HTTP and binary mode connections.

Multiple independent Livy services

When you run more than one Livy service in a cluster (for example, Livy-for-Spark3-1 and Livy-for-Spark3-2), each service must use unique values for livy.server.port, livy.server.recovery.state-store.url, and livy.server.zk.key-prefix. Without unique HDFS recovery paths and ZooKeeper key prefixes, session recovery and leader election can conflict between services.

For step-by-step instructions, see Adding independent Livy services with active/passive HA.