Configure a Spark job for dynamic resource allocation

Use the following steps to configure dynamic resource allocation for a specific job.

There are two ways to customize dynamic resource allocation properties for a specific job:

  • Include property values in the spark-submit command, using the -conf option.

    This approach loads the default spark-defaults.conf file first, and then applies property values specified in your spark-submit command.

    Example:

    spark-submit --conf “property_name=property_value”

  • Create a job-specific spark-defaults.conf file and pass it to the spark-submit command.

    This approach uses the specified properties file, without reading the default property file.

    Example:

    spark-submit --properties-file <property_file>