Configuring Apache Spark
Also available as:
PDF

Dynamic resource allocation properties

The following tables provide more information about dynamic resource allocation properties.

Table 3.1. Dynamic Resource Allocation Properties

Property Name Value Meaning
spark.dynamicAllocation. enabled Default is true for the Spark Thrift server, and false for Spark jobs. Specifies whether to use dynamic resource allocation, which scales the number of executors registered for an application up and down based on workload. Note that this feature is currently only available in YARN mode.
spark.shuffle.service. enabled true

Enables the external shuffle service, which preserves shuffle files written by executors so that the executors can be safely removed.

This property must be set to true if spark.dynamicAllocation. enabledis true.

spark.dynamicAllocation. initialExecutors Default is spark.dynamicAllocation. minExecutors

The initial number of executors to run if dynamic resource allocation is enabled.

This value must be greater than or equal to the minExecutors value, and less than or equal to the maxExecutors value.

spark.dynamicAllocation. maxExecutors Default is infinity Specifies the upper bound for the number of executors if dynamic resource allocation is enabled.
spark.dynamicAllocation. minExecutors Default is 0 Specifies the lower bound for the number of executors if dynamic resource allocation is enabled.

Table 3.2. Optional Dynamic Resource Allocation Properties

Property Name Value Meaning
spark.dynamicAllocation. executorIdleTimeout Default is 60 seconds (60s) If dynamic resource allocation is enabled and an executor has been idle for more than this time, the executor is removed.
spark.dynamicAllocation. cachedExecutorIdleTimeout Default is infinity If dynamic resource allocation is enabled and an executor with cached data blocks has been idle for more than this time, the executor is removed.
spark.dynamicAllocation. schedulerBacklogTimeout 1 second (1s) If dynamic resource allocation is enabled and there have been pending tasks backlogged for more than this time, new executors are requested.
spark.dynamicAllocation. sustainedSchedulerBacklogTimeout Default is schedulerBacklogTimeout Same as spark.dynamicAllocation. schedulerBacklogTimeout,but used only for subsequent executor requests.