Enabling the Livy /ha-status endpoint

The /ha-status endpoint reports whether a Livy server is the leader of its service instance. HAProxy uses it as a health check, so that requests go to the leader instead of to a passive server that answers with an HTTP 307 redirect.

When the endpoint is enabled, GET /ha-status returns the following responses:

Livy server state HTTP status Response body
Leader 200 {"status":"leader"}
Passive (follower) 503 {"status":"follower"}

The endpoint is disabled by default. Livy exposes it only when high availability recovery is active, that is, when livy.server.recovery.mode is ha. Cloudera Manager sets this mode when a Livy service has more than one Livy Server role. If you enable the endpoint while the recovery mode is not ha, Livy logs a warning and does not expose the endpoint.

The endpoint does not require authentication, so that load balancers can probe it without Kerberos credentials. It returns only the leader or follower state of the server.

  1. In Cloudera Manager, select a Livy for Spark 3 service instance and click Configuration.
  2. Search for Livy Server for Spark 3 Advanced Configuration Snippet (Safety Valve) for livy3-conf/livy.conf and add the following property:
    livy.server.ha-status.enabled=true
  3. Click Save Changes.
  4. Repeat these steps for every Livy for Spark 3 service instance behind HAProxy.
  5. Restart the Livy Server roles of each service instance.
  6. Verify the endpoint on each Livy server:
    curl -s -o - -w ' %{http_code}\n' --cacert ca-certificate.pem \
      https://livy-host.example.com:livy-port/ha-status

    Exactly one server in each service instance returns 200; every other server in the same instance returns 503. If a server returns another status, such as 401 or 404, the endpoint is not enabled on that server or the service instance does not use ha recovery mode.