Apache Spark Component Guide
Also available as:
loading table of contents...

Configuring a Job for Dynamic Resource Allocation

There are two ways to customize dynamic resource allocation properties for a specific job:

  • Include property values in the spark-submit command, using the -conf option.

    This approach loads the default spark-defaults.conf file first, and then applies property values specified in your spark-submit command. Here is an example:

    spark-submit --conf “property_name=property_value”

  • Create a job-specific spark-defaults.conf file and pass it to the spark-submit command.

    This approach uses the specified properties file, without reading the default property file. Here is an example:

    spark-submit --properties-file <property_file>