Livy and HAProxy tuning reference

Use these settings as a starting point when many interactive sessions are created at the same time through HAProxy. Adjust the values to the size of your cluster and validate them with your own workload.

HAProxy connection limits and timeouts

The connection limits form a hierarchy: global, defaults, frontend, backend, and server. Low limits cause clients to stall while they wait for a connection. Long timeouts allow slow session creation and deletion to complete.

Setting Example value Where
maxconn 50000 global section
maxconn 5000 defaults section
maxconn 10000 frontend section
maxconn 5000 Default backend that contains all servers
maxconn 2000 Each backend for a service instance
maxconn 800 Each server line
timeout connect 30s defaults section
timeout client, timeout server, timeout http-request 600s defaults section
timeout http-keep-alive 60s defaults section
timeout check 5s defaults section

Use GET /ha-status as the health check of every backend. Do not use GET /sessions as a health check, because it does not identify the leader of a service instance.

Why leader-only routing matters at scale

A passive Livy server answers with an HTTP 307 redirect to the leader. Without /ha-status health checks, a round-robin load balancer sends part of the requests to passive servers. Each redirect adds a request and a Kerberos authentication, can break SRV_ID stickiness, and can cause HTTP 401 or HTTP 504 errors. These failures are rare with few clients but become frequent when hundreds of sessions start at the same time.

Livy server properties

Add the following properties to Livy Server for Spark 3 Advanced Configuration Snippet (Safety Valve) for livy3-conf/livy.conf on every Livy for Spark 3 service instance behind HAProxy:

livy.server.ha-status.enabled=true
livy.server.yarn.poll-interval=5s
livy.server.session.timeout-check=true
livy.server.session.timeout-check.skip-busy=true
livy.server.session.timeout=3600s
livy.server.session.max-creation=700
livy.server.session.state-retain.sec=600
livy.server.yarn.app-lookup-timeout=600s
livy.spark.yarn.queue=livy-yarn-queue
livy.rsc.server.connect.timeout=1200s
livy.rsc.client.connect.timeout=600s
livy.rsc.launcher.port.range=30000~60000
Property Default Example Purpose
livy.server.ha-status.enabled false true Exposes the /ha-status endpoint for HAProxy health checks.
livy.server.session.timeout-check.skip-busy false true Does not time out sessions that are running statements.
livy.server.session.max-creation 100 700 Maximum number of sessions that one Livy server creates at the same time. Set it above the expected number of concurrent session creations.
livy.server.yarn.app-lookup-timeout 120s 600s Time that Livy waits for YARN to report the application of a new session. A busy queue needs more time.
livy.spark.yarn.queue None Queue path Default YARN queue for the sessions of the service instance. See Livy session capacity and YARN queues.
livy.rsc.server.connect.timeout 90s 1200s Time that Livy waits for a new Spark driver to connect back. Drivers start more slowly under load.
livy.rsc.client.connect.timeout 10s 600s Time that Livy waits to connect to a Spark driver.
livy.rsc.launcher.port.range 10000~10010 30000~60000 Ports that Livy uses to communicate with Spark drivers. A narrow range causes BindException errors when many sessions start. The setting applies to the whole Livy server; do not rely on overriding it per session.

The livy.rsc.* properties apply only to interactive sessions; they do not affect batch jobs. Livy ignores property names that it does not recognize without reporting an error, so check the spelling of each property.

Related service settings

Setting Where to set it Purpose
Livy Server Java heap size: 4 GB Livy for Spark 3 service configuration in Cloudera Manager Reduces garbage collection pauses that can expire the ZooKeeper session of the Livy server and cause leader changes.
maxSessionTimeout=300000 ZooKeeper zoo.cfg safety valve Allows longer ZooKeeper sessions, which prevents leader flapping under load.
spark.yarn.am.waitTime=600s Spark 3 spark-defaults.conf safety valve Gives the Spark driver more time to initialize. Without it, slow drivers fail with Application Master exit code 13. Setting this property in livy.conf has no effect.
kdc_tcp_listen_backlog = 128 in the [kdcdefaults] section kdc.conf on each MIT Kerberos KDC of the realm A small TCP listen backlog causes LoginException: Connection reset errors when many clients authenticate at the same time. Restart the KDC after the change. Clients that authenticate in large numbers can also set udp_preference_limit = 1 in krb5.conf to use TCP.

After you change these settings, restart the services in the following order: ZooKeeper, Spark 3, and Livy for Spark 3. If you changed the KDC configuration, restart the KDC. Then reload HAProxy and verify that the HAProxy statistics show one available server per backend.

Client queue and retry settings

Clients should connect through the load balancer URL and retry briefly when a request fails during a leader or gateway failover. For example, a SparkMagic config.json file can set the queue and a retry policy:

{
  "session_configs_defaults": {
    "queue": "livy-yarn-queue"
  },
  "retry_policy": "configurable",
  "retry_seconds_to_sleep_list": [0.2, 0.5, 1, 3, 5],
  "configurable_retry_policy_max_retries": 8
}

Reference results

In Cloudera testing with six Livy service instances behind HAProxy, 400 concurrent interactive sessions were created, used, and deleted without errors after all of the settings on this page were applied. These results show the effect of the tuning; they are not a performance guarantee.