Improving Software Performance

To improve Spark performance, assess and tune the following operations:

  • Minimize shuffle operations where possible.

  • Match join strategy (SortMergeJoin vs. BroadcastHashJoin) to the table size. Spark SQL often uses sort-merge join for large tables and broadcast join when one side is small enough to broadcast.

    This requires manual configuration.

  • Consider switching from the default serializer to the Kryo serializer to improve performance.

    This requires manual configuration and class registration.

  • Configure Spark executor sizing on YARN

Configure Spark executor sizing on YARN

This section describes how to size Spark executors (count, memory, and cores) for applications that run on YARN, based on node hardware and YARN capacity.

YARN evaluates all available compute resources on each machine in a cluster and negotiates resource requests from applications running in the cluster. YARN then provides processing capacity to each application by allocating containers. A container is the basic unit of processing capacity in YARN; it is an encapsulation of resource elements such as memory (RAM) and CPU.

In a Hadoop cluster, it is important to balance the use of RAM, CPU cores, and disks so that processing is not constrained by any one of these cluster resources.

When sizing Spark executors on YARN, note the following values on each node:

  • RAM (amount of memory)

  • CORES (number of CPU cores)

When configuring executor sizing for Spark on YARN, consider the following information:

  • Driver memory does not need to be large if the job does not aggregate much data (as with a collect() action).

  • There are tradeoffs between num-executors and executor-memory.

    Large executor memory does not imply better performance, due to JVM garbage collection. Sometimes it is better to configure a larger number of small JVMs than a small number of large JVMs.

  • Executor processes are not released if the job has not finished, even if they are no longer in use.

    Therefore, do not overallocate executors above your estimated requirements.

For example, on hosts with 64 GB of RAM and 16 cores, you might set yarn.nodemanager.resource.memory-mb to 64512 (63 GB) and yarn.nodemanager.resource.cpu-vcores to 15, leaving capacity for the operating system and Hadoop daemons. If you run three executors per host, divide the per-host memory among them: 63 GB divided by 3 is about 21 GB per executor heap before overhead. Subtract executor memory overhead (described in the Tuning Resource Allocation topic) to choose --executor-memory, for example about 19G with five cores per executor. Scale --num-executors to the cluster. For a full walkthrough including ApplicationMaster placement, see the Resource Tuning Example topic.

On CDP clusters, dynamic allocation is enabled by default. If you set --num-executors (or spark.executor.instances) to fix the executor count, disable dynamic allocation for that application. Alternatively, omit --num-executors and tune dynamic allocation properties instead.

In cluster deploy mode (--deploy-mode cluster) with --master yarn, the Spark driver runs inside an application master process that is managed by YARN on the cluster. The client can stop after initiating the application. The following example submits SparkPi in cluster mode, specifying the number of executors and associated memory and cores, and driver memory. Dynamic allocation and the external shuffle service are disabled so --num-executors controls the executor count. The client starts the default Application Master, and SparkPi runs as a child thread of the Application Master. The client periodically polls the Application Master for status updates and displays them on the console.

spark-submit \
    --master yarn \
    --deploy-mode cluster \
    --num-executors 3 \
    --driver-memory 2g \
    --executor-memory 2g \
    --executor-cores 1 \
    --conf spark.dynamicAllocation.enabled=false \
    --conf spark.shuffle.service.enabled=false \
    --class org.apache.spark.examples.SparkPi \
    $SPARK_HOME/examples/jars/spark-examples*.jar 1000

In client deploy mode (--deploy-mode client) with --master yarn, the driver runs in the client process. The application master is used only to request resources from YARN. The following example launches the Spark shell in client mode with a fixed executor count and the same dynamic allocation settings:

spark-shell \
    --master yarn \
    --deploy-mode client \
    --num-executors 3 \
    --driver-memory 2g \
    --executor-memory 2g \
    --executor-cores 1 \
    --conf spark.dynamicAllocation.enabled=false \
    --conf spark.shuffle.service.enabled=false