Configuring a Job for Dynamic Resource Allocation
There are two ways to customize dynamic resource allocation properties for a specific job:
Include property values in the
spark-submit
command, using the-conf
option.This approach loads the default
spark-defaults.conf
file first, and then applies property values specified in yourspark-submit
command. Here is an example:spark-submit —conf “property_name=property_value”
Create a job-specific
spark-defaults.conf
file and pass it to thespark-submit
command.This approach uses the specified properties file, without reading the default property file. Here is an example:
spark-submit —properties-file <property_file>