Configuring Apache Spark

Also available as:

PDF

loading table of contents...

Dynamic resource allocation properties

The following tables provide more information about dynamic resource allocation properties.

Table 3.1. Dynamic Resource Allocation Properties


Property Name	Value	Meaning
`spark.dynamicAllocation. enabled`	Default is `true` for the Spark Thrift server, and `false` for Spark jobs.	Specifies whether to use dynamic resource allocation, which scales the number of executors registered for an application up and down based on workload. Note that this feature is currently only available in YARN mode.
`spark.shuffle.service. enabled`	`true`	Enables the external shuffle service, which preserves shuffle files written by executors so that the executors can be safely removed. This property must be set to `true` if `spark.dynamicAllocation. enabled`is `true`.
`spark.dynamicAllocation. initialExecutors`	Default is `spark.dynamicAllocation. minExecutors`	The initial number of executors to run if dynamic resource allocation is enabled. This value must be greater than or equal to the `minExecutors` value, and less than or equal to the `maxExecutors` value.
`spark.dynamicAllocation. maxExecutors`	Default is infinity	Specifies the upper bound for the number of executors if dynamic resource allocation is enabled.
`spark.dynamicAllocation. minExecutors`	Default is 0	Specifies the lower bound for the number of executors if dynamic resource allocation is enabled.

Table 3.2. Optional Dynamic Resource Allocation Properties


Property Name	Value	Meaning
`spark.dynamicAllocation. executorIdleTimeout`	Default is 60 seconds (`60s`)	If dynamic resource allocation is enabled and an executor has been idle for more than this time, the executor is removed.
`spark.dynamicAllocation. cachedExecutorIdleTimeout`	Default is infinity	If dynamic resource allocation is enabled and an executor with cached data blocks has been idle for more than this time, the executor is removed.
`spark.dynamicAllocation. schedulerBacklogTimeout`	1 second (`1s`)	If dynamic resource allocation is enabled and there have been pending tasks backlogged for more than this time, new executors are requested.
`spark.dynamicAllocation. sustainedSchedulerBacklogTimeout`	Default is `schedulerBacklogTimeout`	Same as `spark.dynamicAllocation. schedulerBacklogTimeout,`but used only for subsequent executor requests.