Livy performance best practices

Run sessions in YARN cluster mode

Run Apache Livy sessions in YARN cluster mode, so that YARN accounts for the resources of each session and Spark drivers do not run on the Livy server host. For more information, see the Apache Livy Getting Started documentation.

Size the Livy server and limit concurrency

The minimum heap size for the Livy server is rarely enough for production workloads with many concurrent sessions. Increase the heap size of the Livy server based on the number of sessions and statements that you expect.

Set operational limits for:

• The number of sessions that each user can keep open.
• The number of concurrent statements in each session.
• The total number of active sessions. The livy.server.session.max-creation property limits how many sessions can be starting at the same time.

Make session lifetimes intentional

Frequent session timeouts cause users to create sessions again and again, which adds load to YARN and to the Livy server.

• Set the session time-to-live (ttl) for each workload when you create the session. For more information, see the Apache Livy REST API documentation.
• Disable the session timeout check only if you also have a process that stops unused sessions, because these sessions are not terminated automatically. For more information, see Configuring Indefinite Sessions for Livy.

Isolate interactive workloads

Use a dedicated YARN queue for notebooks and Livy sessions, with guaranteed minimum resources, so that batch jobs do not starve interactive sessions.

Start sessions in advance for predictable peaks

If many users open notebooks at the same time every day, the peak in session creation is predictable. You can start a small number of sessions before the peak and reuse them. Keep the default number of executors small, and let dynamic allocation add executors after the first statement. For more information, see Dynamic allocation.

Keep Spark workloads efficient

Apache Spark tuning often gives the biggest improvement in Livy performance:

• Avoid many small files, control the number of shuffle partitions, and use the right join strategies.
• Use the Spark UI as your main tool to profile slow statements.
• Keep the Spark configuration stable. List the Spark properties that users must not override in the Livy list of blocked Spark properties, the spark-blacklist.conf file.

Configure high availability and session recovery carefully

If you run Livy with active/passive high availability:

• Ensure that Apache ZooKeeper is healthy and has low latency.
• Ensure that the recovery state store is fast and reliable.
• Stop sessions that are no longer needed. Every session that is still known to the state store is recovered when a Livy server restarts, which can exhaust the Livy server heap immediately after the restart.

For more information, see Livy active/passive high availability. To diagnose slowness, see Troubleshooting for Livy slowness.