Configure a Spark job for dynamic resource allocation

There are two ways to customize dynamic resource allocation properties for a specific job:

• Include property values in the spark-submit command, using the -conf option.

  This approach loads the default spark-defaults.conf file first, and then applies property values specified in your spark-submit command.

  Example:

  spark-submit --conf “property_name=property_value”

• Create a job-specific spark-defaults.conf file and pass it to the spark-submit command.

  This approach uses the specified properties file, without reading the default property file.

  Example:

  spark-submit --properties-file <property_file>