Livy session capacity and YARN queues

Adding Livy service instances behind HAProxy increases the number of sessions that Livy can manage, but every session also needs YARN resources. The YARN queue that runs the sessions often limits concurrency before Livy does.

Application Master resource limit

Each interactive Livy session runs a Spark driver in a YARN Application Master (AM) container. The Capacity Scheduler limits the share of a queue that AM containers can use with yarn.scheduler.capacity.queue-path.maximum-am-resource-percent. When AM containers reach this limit, new sessions stay in the ACCEPTED state in YARN and in the starting state in Livy, even if the cluster has free memory.

For example, if the sessions run in a queue with 16 GB of resources and the AM limit of the queue is 0.2, AM containers can use 3.2 GB in total.

Container size of a session

YARN allocates more memory than the requested driver memory:

  1. Spark adds memory overhead to the driver memory. The overhead is at least 384 MB.
  2. YARN rounds the request up to a multiple of yarn.scheduler.minimum-allocation-mb.

For example, a session with 1024 MB of driver memory requests 1024 MB + 384 MB = 1408 MB. If yarn.scheduler.minimum-allocation-mb is 1024 MB, YARN allocates a 2048 MB container. With the 3.2 GB AM limit from the previous example, only one such session can start; the second one would raise AM usage to 4 GB.

To estimate the number of concurrent sessions in a queue, divide the AM resources of the queue by the rounded container size of one session.

Resource calculator

The Capacity Scheduler resource calculator determines which resources YARN counts:

  • DefaultResourceCalculator counts only memory. More sessions can start, but drivers can stay in the starting state when the hosts run out of CPU.
  • DominantResourceCalculator counts memory and vCores. A session starts only when both are available, which avoids session start timeouts caused by CPU contention.

Using a dedicated queue for Livy sessions

Cloudera recommends running Livy sessions in a dedicated YARN queue, so that interactive sessions do not compete with batch jobs and the limits are predictable. Size the capacity, maximum-applications, and maximum-am-resource-percent properties of that queue for the number of concurrent sessions that you expect across all Livy service instances.

The following Capacity Scheduler properties show an example with a child queue named livy:

<property>
  <name>yarn.scheduler.capacity.root.queues</name>
  <value>default,livy</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.default.capacity</name>
  <value>20</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.livy.capacity</name>
  <value>80</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.livy.maximum-applications</name>
  <value>1200</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.livy.maximum-am-resource-percent</name>
  <value>0.8</value>
</property>

The capacities of the child queues of a parent queue must add up to 100%, and all queues at the same level must use the same capacity mode. Apply the properties in the Capacity Scheduler configuration snippet of the YARN service or with the YARN Queue Manager, and then restart or refresh the ResourceManager as Cloudera Manager prompts. For more information, see Setting Application-Master resource-limit for a specific queue.

How Livy selects the queue

Livy submits a session to the first queue that is set in the following order:

  1. The queue field of the session or batch request.
  2. The livy.spark.yarn.queue property in the Livy server configuration, which sets a default queue for all sessions of a Livy service instance.
  3. The spark.yarn.queue property in the conf map of the request.
  4. The default queue of the cluster.

Set livy.spark.yarn.queue to the same value on every Livy service instance behind HAProxy, and use the full queue path that the ResourceManager reports, for example root.livy.