Cloudera Docs

Troubleshooting for Livy slowness

Slow Livy session creation, statement execution, or REST API responses usually come from YARN resource shortages, inefficient Spark workloads, an overloaded Livy server, or slow HDFS storage.

Condition

Users report that Apache Livy is slow. This usually means one or more of the following:

  • Slow session creation: POST /sessions takes minutes, and sessions remain in the starting or not_started state.
  • Slow statement execution: POST /sessions/{id}/statements returns quickly, but the statement remains in the waiting or running state for longer than expected.
  • Slow REST API responses: calls such as GET /sessions, GET /sessions/{id}, or GET /sessions/{id}/log are slow or time out.

Identify the source of the latency

Follow these checks in order each time Livy slowness is reported. They narrow the cause to one of the scenarios described in the rest of this topic.

  1. Identify the kind of latency. Record the time the client sent the request, the time Livy responded, the session ID and statement ID, the Livy server host that handled the request (if Livy runs behind a load balancer or Apache Knox), and the user or proxy user that submitted the request.

  2. Check the state that Livy reports. Use the Livy REST API:

    • GET /sessions: check how many sessions are active.
    • GET /sessions/{id}: check the session state and its transitions.
    • GET /sessions/{id}/log: check startup, failure, and waiting messages.

    If a session remains in the starting state, investigate YARN scheduling and the Apache Spark submission path. If a statement remains in the waiting or running state, open the Spark UI for that application. For more information about the endpoints, see Livy API reference for interactive sessions and the Apache Livy REST API documentation.

  3. Correlate with the YARN and Spark UIs. For the YARN application of the session, check how long the application stays in each state (SUBMITTED, ACCEPTED, RUNNING). A long time in ACCEPTED means that the application is waiting for its ApplicationMaster container. In the Spark UI, check whether the driver is alive, how quickly executors are added, and whether stages show skew, large shuffles, spills, or long garbage collection (GC) times.

    In general:

    • If the application waits to start, the cause is in YARN or cluster resources.
    • If the application runs but its stages are slow, the cause is in the Spark workload.
    • If the YARN and Spark UIs look normal but Livy endpoints are slow, the cause is on the Livy server.

  4. Check the health of the Livy server host. Check CPU load, free memory, swap usage, and disk I/O for the log and temporary directories. Check the Livy server JVM for long GC pauses, and collect thread dumps while the slowness is happening.

  5. Check storage and node health. Look for HDFS read latency spikes, executor shuffle fetch wait time spikes, and NodeManager disk health alerts. For more information, see Detecting slow DataNodes.

  6. Check high availability and session recovery. If Livy runs with more than one Livy server or with session recovery enabled, check Apache ZooKeeper latency and session expirations, the latency of the recovery state store, and whether slowness started immediately after a Livy server restart. For more information, see Livy active/passive high availability.

  7. Search the logs for the slow period. In Cloudera Manager, search the Livy server logs for the time window of the slowness. Look for repeated session retries, authentication failures and retries, long session creation times, and GC, OutOfMemoryError, or thread pool warnings. For more information, see Logs.

Session creation takes 10 to 20 minutes

The YARN queue is at or over capacity, so the ApplicationMaster container of the session is not allocated. Not enough free memory or vcores, unhealthy or decommissioning NodeManagers, full disks, or preemption can also delay container allocation. The session remains in the starting state, and the YARN application remains in the ACCEPTED state.

Solution

  • Guarantee resources for interactive workloads, for example, with a dedicated Capacity Scheduler queue for Livy sessions.
  • Reduce the default driver and executor sizes so that sessions can start when the cluster is under pressure, and let dynamic allocation add executors after the first statement.
  • If many users open notebooks at the same time, start a small number of sessions in advance and reuse them.

All Livy endpoints are slow for every user

The Livy server JVM is struggling. Common causes are long GC pauses because the Livy server heap is too small for the number of sessions, CPU starvation, thread contention from many concurrent sessions and statements, and disk I/O pressure from heavy logging. Authentication overhead (TLS and Kerberos) increases under load. Even simple calls such as GET /sessions are slow.

The Livy server heap can also fill up immediately after a restart when session recovery is enabled and the server recovers a large number of sessions, especially batch sessions, from the state store.

Solution

  • Increase the Livy server heap size and tune garbage collection. The G1 garbage collector is usually a good choice.
  • Reduce logging volume during normal operation.
  • Run Spark sessions in YARN cluster mode so that Spark drivers do not run on the Livy server host.
  • Limit the number of concurrent sessions. The livy.server.session.max-creation property limits how many sessions can be starting at the same time.
  • Stop sessions that are no longer needed, so that they are not recovered after each restart.

Only some statements are slow

The Spark query itself is slow. Common causes are data skew, an inefficient join strategy, very large shuffles, too few shuffle partitions, many small files, and Apache Hive Metastore latency, which delays the start of SQL queries. The statement remains in the running state, and the Spark UI shows active stages for a long time.

Solution

  • Enable Adaptive Query Execution (AQE) for Spark SQL.
  • Fix data skew, for example, with salting, skew join hints, or repartitioning.
  • Tune the number of shuffle partitions and the executor sizes.

Livy is randomly slow at times

Some queries are fast and others are suddenly slow, because Spark tasks wait on I/O. Common causes are slow DataNodes (partial failures of disks, network interface cards, or controllers, or poor operating system and JVM tuning) and network congestion between compute and storage. Task I/O times spike, and HDFS reports slow nodes.

Solution

  • Use slow DataNode detection to find the affected nodes, then fix the node by replacing failing disks, network interface cards, or controllers. For more information, see Detecting slow DataNodes.
  • Rebalance data and fix operating system tuning on the affected nodes.

Requests are slow after a failover or when a Livy server is stopped

With active/passive high availability, passive Livy servers redirect requests to the active server. If one of the Livy servers is stopped, or ZooKeeper or the recovery state store is slow, requests can be delayed before they reach the active server.

Solution

  • Ensure that all Livy server roles are running, or remove the roles that you do not use.
  • Ensure that ZooKeeper is healthy and has low latency, and that the recovery state store is fast and reliable.

To prevent Livy slowness, see Livy performance best practices.