Configuring Dynamic Resource Allocation
This section describes how to configure dynamic resource allocation for Apache Spark.
When the dynamic resource allocation feature is enabled, an application's use of executors is dynamically adjusted based on workload. This means that an application can relinquish resources when the resources are no longer needed, and request them later when there is more demand. This feature is particularly useful if multiple applications share resources in your Spark cluster.
Dynamic resource allocation is available for use by the Spark Thrift server and general Spark jobs.
 | Note |
|---|---|
Dynamic Resource Allocation does not work with Spark Streaming. |
You can configure dynamic resource allocation at either the cluster or the job level:
-
Cluster level:
-
On an Ambari-managed cluster, the Spark Thrift server uses dynamic resource allocation by default. The Thrift server increases or decreases the number of running executors based on a specified range, depending on load. (In addition, the Thrift server runs in YARN mode by default, so the Thrift server uses resources from the YARN cluster.) The associated shuffle service starts automatically, for use by the Thrift server and general Spark jobs.
-
On a manually installed cluster, dynamic resource allocation is not enabled by default for the Thrift server or for other Spark applications. You can enable and configure dynamic resource allocation and start the shuffle service during the Spark manual installation or upgrade process.
-
-
Job level: You can customize dynamic resource allocation settings on a per-job basis. Job settings override cluster configuration settings.
Cluster configuration is the default, unless overridden by job configuration.
The following subsections describe each configuration approach, followed by a list of dynamic resource allocation properties and a set of instructions for customizing the Spark Thrift server port.