Livy and HAProxy tuning reference
Use these settings as a starting point when many interactive sessions are created at the same time through HAProxy. Adjust the values to the size of your cluster and validate them with your own workload.
HAProxy connection limits and timeouts
The connection limits form a hierarchy: global, defaults, frontend, backend, and server. Low limits cause clients to stall while they wait for a connection. Long timeouts allow slow session creation and deletion to complete.
| Setting | Example value | Where |
|---|---|---|
maxconn |
50000 | global section |
maxconn |
5000 | defaults section |
maxconn |
10000 | frontend section |
maxconn |
5000 | Default backend that contains all servers |
maxconn |
2000 | Each backend for a service instance |
maxconn |
800 | Each server line |
timeout connect |
30s | defaults section |
timeout client, timeout server,
timeout http-request |
600s | defaults section |
timeout http-keep-alive |
60s | defaults section |
timeout check |
5s | defaults section |
Use GET /ha-status as the health check of every backend. Do not use
GET /sessions as a health check, because it does not identify the leader
of a service instance.
Why leader-only routing matters at scale
A passive Livy server answers with an HTTP 307 redirect to the leader.
Without /ha-status health checks, a round-robin load balancer sends part
of the requests to passive servers. Each redirect adds a request and a Kerberos
authentication, can break SRV_ID stickiness, and can cause
HTTP 401 or HTTP 504 errors. These failures are rare
with few clients but become frequent when hundreds of sessions start at the same
time.
Livy server properties
Add the following properties to Livy Server for Spark 3 Advanced Configuration Snippet (Safety Valve) for livy3-conf/livy.conf on every Livy for Spark 3 service instance behind HAProxy:
livy.server.ha-status.enabled=true
livy.server.yarn.poll-interval=5s
livy.server.session.timeout-check=true
livy.server.session.timeout-check.skip-busy=true
livy.server.session.timeout=3600s
livy.server.session.max-creation=700
livy.server.session.state-retain.sec=600
livy.server.yarn.app-lookup-timeout=600s
livy.spark.yarn.queue=livy-yarn-queue
livy.rsc.server.connect.timeout=1200s
livy.rsc.client.connect.timeout=600s
livy.rsc.launcher.port.range=30000~60000
| Property | Default | Example | Purpose |
|---|---|---|---|
livy.server.ha-status.enabled |
false | true | Exposes the /ha-status endpoint for HAProxy health
checks. |
livy.server.session.timeout-check.skip-busy |
false | true | Does not time out sessions that are running statements. |
livy.server.session.max-creation |
100 | 700 | Maximum number of sessions that one Livy server creates at the same time. Set it above the expected number of concurrent session creations. |
livy.server.yarn.app-lookup-timeout |
120s | 600s | Time that Livy waits for YARN to report the application of a new session. A busy queue needs more time. |
livy.spark.yarn.queue |
None | Queue path | Default YARN queue for the sessions of the service instance. See Livy session capacity and YARN queues. |
livy.rsc.server.connect.timeout |
90s | 1200s | Time that Livy waits for a new Spark driver to connect back. Drivers start more slowly under load. |
livy.rsc.client.connect.timeout |
10s | 600s | Time that Livy waits to connect to a Spark driver. |
livy.rsc.launcher.port.range |
10000~10010 | 30000~60000 | Ports that Livy uses to communicate with Spark drivers. A narrow range causes
BindException errors when many sessions start. The setting applies
to the whole Livy server; do not rely on overriding it per session. |
The livy.rsc.* properties apply only to interactive sessions; they do not
affect batch jobs. Livy ignores property names that it does not recognize without reporting
an error, so check the spelling of each property.
Related service settings
| Setting | Where to set it | Purpose |
|---|---|---|
| Livy Server Java heap size: 4 GB | Livy for Spark 3 service configuration in Cloudera Manager | Reduces garbage collection pauses that can expire the ZooKeeper session of the Livy server and cause leader changes. |
maxSessionTimeout=300000 |
ZooKeeper zoo.cfg safety valve | Allows longer ZooKeeper sessions, which prevents leader flapping under load. |
spark.yarn.am.waitTime=600s |
Spark 3 spark-defaults.conf safety valve | Gives the Spark driver more time to initialize. Without it, slow drivers fail with Application Master exit code 13. Setting this property in livy.conf has no effect. |
kdc_tcp_listen_backlog = 128 in the
[kdcdefaults] section |
kdc.conf on each MIT Kerberos KDC of the realm | A small TCP listen backlog causes LoginException: Connection
reset errors when many clients authenticate at the same time. Restart the
KDC after the change. Clients that authenticate in large numbers can also set
udp_preference_limit = 1 in krb5.conf to use
TCP. |
After you change these settings, restart the services in the following order: ZooKeeper, Spark 3, and Livy for Spark 3. If you changed the KDC configuration, restart the KDC. Then reload HAProxy and verify that the HAProxy statistics show one available server per backend.
Client queue and retry settings
Clients should connect through the load balancer URL and retry briefly when a request fails during a leader or gateway failover. For example, a SparkMagic config.json file can set the queue and a retry policy:
{
"session_configs_defaults": {
"queue": "livy-yarn-queue"
},
"retry_policy": "configurable",
"retry_seconds_to_sleep_list": [0.2, 0.5, 1, 3, 5],
"configurable_retry_policy_max_retries": 8
}
Reference results
In Cloudera testing with six Livy service instances behind HAProxy, 400 concurrent interactive sessions were created, used, and deleted without errors after all of the settings on this page were applied. These results show the effect of the tuning; they are not a performance guarantee.
