Finding the Livy server that owns a session

Behind HAProxy, all Livy servers share one URL. To connect another client, such as Beeline, to an existing interactive session, find the Livy server that owns the session from the livyHost field and map it to the matching SRV_ID cookie.

For interactive sessions, Livy adds a livyHost entry to the appInfo map of the session. The entry contains the host name of the Livy server that answered the request, which is the leader of the backend that owns the session.

  1. Retrieve the session through the load balancer, using the cookie file of the client that created the session:
    curl -sk --negotiate -u : -c cookies.dat -b cookies.dat \
      https://livy-lb.example.com:8888/sessions/session-id | jq '.appInfo'

    The output contains the livyHost entry, for example:

    {
      "driverLogUrl": "...",
      "sparkUiUrl": "...",
      "livyHost": "livy1-a.example.com"
    }
  2. Find the server line for that host in haproxy.cfg and note its cookie value.

    For example, the line server livy1-a livy1-a.example.com:28998 ... cookie s1 means that the session belongs to SRV_ID=s1. Depending on the host configuration, livyHost can contain a short host name or an FQDN. Client tools that connect often can keep the host-to-cookie mapping in their own configuration.

  3. Send the cookie with every request that must reach the session. For example, connect Beeline to the session through the Livy Thrift server:
    beeline -u "jdbc:hive2://livy-lb.example.com:haproxy-thrift-port/;principal=HTTP/livy-lb.example.com@REALM;ssl=true;sslTrustStore=truststore.jks;trustStorePassword=truststore-password;trustStoreType=jks;transportMode=http;httpHeader.Cookie=SRV_ID=s1;httpPath=cliservice?livy.session.name=session-name"

    For details about the connection parameters, see Connecting to the Livy Thrift server through HAProxy.

Requests that carry the SRV_ID cookie reach the backend that owns the session. If the leader of that backend changes, HAProxy sends the requests to the new leader in the same backend.