Spark Guide
Also available as:
PDF
loading table of contents...

Customizing Cluster Dynamic Resource Allocation Settings (Ambari)

On an Ambari-managed cluster, dynamic resource allocation is configured for the Spark Thrift Server as part of the Spark installation process. Ambari starts the shuffle service for use by the Thrift Server and general Spark jobs.

To view or modify property values for the Spark Thrift Server, navigate to Services > Spark. Required settings are listed in the "Advanced spark-thrift-sparkconf" group; additional properties can be specified in the custom section. (For a complete list of DRA properties, see Dynamic Resource Allocation Properties.)

To enable and configure dynamic resource allocation for general Spark applications, navigate to Services > Spark. Review the list of properties in the Advanced spark-defaults group, and revise settings as needed.

Dynamic resource allocation requires an external shuffle service on each worker node. If you installed your cluster using Ambari, the service will be started automatically; no further steps are needed.