Spark Guide
Also available as:
PDF
loading table of contents...

Configuring a Job for Dynamic Resource Allocation

There are two ways to customize dynamic resource allocation properties for a specific job:

  • Include property values in the spark-submit command, using the -conf option.

    This approach will load the default spark-defaults.conf file first, and then apply property values specified in your spark-submit command. Here is an example:

    spark-submit —conf “property_name=property_value”

  • Create a job-specific spark-defaults.conf file and pass it to the spark-submit command.

    This approach will use the specified properties-file, without reading the default property file.

    spark-submit —properties-file <property_file>