Apache Spark Component Guide
Also available as:
PDF
loading table of contents...

Dynamic Resource Allocation Properties

See the following tables for more information about basic and optional dynamic resource allocation properties. For more information, see the Apache Dynamic Resource Allocation documentation.

Table 3.1. Dynamic Resource Allocation Properties

Property NameValueMeaning
spark.dynamicAllocation. enabledDefault is true for the Spark Thrift server, and false for Spark jobs.

Specifies whether to use dynamic resource allocation, which scales the number of executors registered for an application up and down based on workload. Note that this feature is currently only available in YARN mode.

spark.shuffle.service. enabledtrue

Enables the external shuffle service, which preserves shuffle files written by executors so that the executors can be safely removed.

This property must be set to true if spark.dynamicAllocation. enabled is true.

spark.dynamicAllocation. initialExecutorsDefault is spark.dynamicAllocation. minExecutors

The initial number of executors to run if dynamic resource allocation is enabled.

This value must be greater than or equal to the minExecutors value, and less than or equal to the maxExecutors value.

spark.dynamicAllocation. maxExecutorsDefault is infinity

Specifies the upper bound for the number of executors if dynamic resource allocation is enabled.

spark.dynamicAllocation. minExecutorsDefault is 0

Specifies the lower bound for the number of executors if dynamic resource allocation is enabled.


Table 3.2. Optional Dynamic Resource Allocation Properties

Property NameValueMeaning
spark.dynamicAllocation. executorIdleTimeoutDefault is 60 seconds (60s)

If dynamic resource allocation is enabled and an executor has been idle for more than this time, the executor is removed.

spark.dynamicAllocation. cachedExecutorIdleTimeoutDefault is infinity

If dynamic resource allocation is enabled and an executor with cached data blocks has been idle for more than this time, the executor is removed.

spark.dynamicAllocation. schedulerBacklogTimeout1 second (1s)

If dynamic resource allocation is enabled and there have been pending tasks backlogged for more than this time, new executors are requested.

spark.dynamicAllocation. sustainedSchedulerBacklogTimeoutDefault is schedulerBacklogTimeout

Same as spark.dynamicAllocation. schedulerBacklogTimeout,but used only for subsequent executor requests.