Configuring Dynamic Resource Allocation and Thrift Server Settings
When the dynamic resource allocation feature is enabled, an application's use of executors is dynamically adjusted based on workload. This means that an application may relinquish resources when the resources are no longer needed, and request them later when there is more demand. This feature is particularly useful if multiple applications share resources in your Spark cluster.
Dynamic resource allocation is available for use by the Spark Thrift Server and general Spark jobs. You can configure dynamic resource allocation at the cluster or job level:
On an Ambari-managed cluster, the Spark Thrift Server uses dynamic resource allocation by default. The Thrift Server will increase or decrease the number of running executors based on a specified range, depending on load. (In addition, the Thrift Server runs in YARN mode by default, so the Thrift Server will use resources from the YARN cluster.)
On a manually-installed cluster, dynamic resource allocation is not enabled by default for the Thrift Server or for other Spark applications. You can enable and configure dynamic resource allocation and start the shuffle service during the Spark manual installation or upgrade process.
You can customize dynamic resource allocation settings on a per-job basis. Job settings will override cluster configuration settings.
Cluster configuration is the default unless overridden by job configuration.
The next three subsections describe each configuration approach, followed by a list of dynamic resource allocation properties and a set of instructions for customizing the Spark Thrift Server port.