Configuring a Job for Dynamic Resource Allocation
There are two ways to customize dynamic resource allocation properties for a specific job:
Include property values in the
spark-submit
command, using the-conf
option.This approach will load the default
spark-defaults.conf
file first, and then apply property values specified in yourspark-submit
command. Here is an example:spark-submit —conf “property_name=property_value”
Create a job-specific
spark-defaults.conf
file and pass it to thespark-submit
command.This approach will use the specified properties-file, without reading the default property file.
spark-submit —properties-file <property_file>