Dynamic resource allocation properties

The following tables provide more information about dynamic resource allocation properties. These properties can be accessed by clicking through to Cloudera Manager from the Cloudera Data Platform interface for the cluster running Spark.

Table 1. Dynamic Resource Allocation Properties
Property Name Default Value Description
spark.dynamicAllocation.enabled false for Spark jobs Enable dynamic allocation of executors in Spark applications.
spark.shuffle.service. enabled true Enables the external shuffle service. The external shuffle service preserves shuffle files written by executors so that the executors can be deallocated without losing work. Must be enabled if dynamic allocation is enabled.
spark.dynamicAllocation. initialExecutors Same value as spark.dynamicAllocation. minExecutors

When dynamic allocation is enabled, number of executors to allocate when the application starts.

Must be greater than or equal to the minExecutors value, and less than or equal to the maxExecutors value.

spark.dynamicAllocation. maxExecutors infinity When dynamic allocation is enabled, maximum number of executors to allocate. By default, Spark relies on YARN to control the maximum number of executors for the application.
spark.dynamicAllocation. minExecutors 0 When dynamic allocation is enabled, minimum number of executors to keep alive while the application is running.
spark.dynamicAllocation. executorIdleTimeout 60 seconds (60s) When dynamic allocation is enabled, time after which idle executors will be stopped.
spark.dynamicAllocation. cachedExecutorIdleTimeout infinity When dynamic allocation is enabled, time after which idle executors with cached RDD blocks will be stopped.
spark.dynamicAllocation. schedulerBacklogTimeout 1 second (1s) When dynamic allocation is enabled, timeout before requesting new executors when there are backlogged tasks.
spark.dynamicAllocation. sustainedSchedulerBacklogTimeout Same value as schedulerBacklogTimeout When dynamic allocation is enabled, timeout before requesting new executors after the initial backlog timeout has already expired. By default this is the same value as the initial backlog timeout.