Whether to use dynamic resource allocation, which scales the number of executors registered with this application up and down based on the workload. Note that this is currently only available in YARN mode. For more information, see the Apache Dynamic Resource Allocation documentation.

DRA requires spark.shuffle.service.enabled to be set. The following configurations are also relevant: spark.dynamicAllocation. minExecutors, spark.dynamicAllocation. maxExecutors, and spark.dynamicAllocation. initialExecutors

Enables the external shuffle service, which preserves shuffle files written by executors so that the executors can be safely removed.

This property must be set to true if spark.dynamicAllocation. enabled is true.

The external shuffle service must be set up before enabling the property. For more information, see "Starting the Shuffle Service" at the end of this section.

Initial number of executors to run if dynamic resource allocation is enabled. This value must be greater than or equal to the minExecutors value, and less than or equal to the maxExecutors value.

Upper bound for the number of executors if dynamic resource allocation is enabled.

Lower bound for the number of executors if dynamic resource allocation is enabled.

If dynamic resource allocation is enabled and an executor has been idle for more than this duration, the executor will be removed.

If dynamic resource allocation is enabled and an executor with cached data blocks has been idle for more than this duration, the executor will be removed.

If dynamic resource allocation is enabled and there have been pending tasks backlogged for more than this duration, new executors will be requested.

Same as spark.dynamicAllocation. schedulerBacklogTimeout,but used only for subsequent executor requests.