Dynamic resource allocation properties

The following tables provide more information about dynamic resource allocation properties. These properties can be accessed by clicking through to Cloudera Manager from the Cloudera Data Platform interface for the cluster running Spark.

Table 1. Dynamic Resource Allocation Properties
Property Name	Default Value	Description
`spark.dynamicAllocation.enabled`	`false` for Spark jobs	Enable dynamic allocation of executors in Spark applications.
`spark.shuffle.service. enabled`	`true`	Enables the external shuffle service. The external shuffle service preserves shuffle files written by executors so that the executors can be deallocated without losing work. Must be enabled if dynamic allocation is enabled.
`spark.dynamicAllocation. initialExecutors`	Same value as `spark.dynamicAllocation. minExecutors`	When dynamic allocation is enabled, number of executors to allocate when the application starts. Must be greater than or equal to the `minExecutors` value, and less than or equal to the `maxExecutors` value.
`spark.dynamicAllocation. maxExecutors`	infinity	When dynamic allocation is enabled, maximum number of executors to allocate. By default, Spark relies on YARN to control the maximum number of executors for the application.
`spark.dynamicAllocation. minExecutors`	0	When dynamic allocation is enabled, minimum number of executors to keep alive while the application is running.
`spark.dynamicAllocation. executorIdleTimeout`	60 seconds (`60s`)	When dynamic allocation is enabled, time after which idle executors will be stopped.
`spark.dynamicAllocation. cachedExecutorIdleTimeout`	infinity	When dynamic allocation is enabled, time after which idle executors with cached RDD blocks will be stopped.
`spark.dynamicAllocation. schedulerBacklogTimeout`	1 second (`1s`)	When dynamic allocation is enabled, timeout before requesting new executors when there are backlogged tasks.
`spark.dynamicAllocation. sustainedSchedulerBacklogTimeout`	Same value as `schedulerBacklogTimeout`	When dynamic allocation is enabled, timeout before requesting new executors after the initial backlog timeout has already expired. By default this is the same value as the initial backlog timeout.