Submitting Spark applications
To submit an application consisting of a Python file or a
compiled and packaged Java or Spark JAR, use the
spark-submit script.
spark-submit Syntax
spark-submit --option value \
application jar | python file [application arguments]
Example: Running SparkPi on YARN demonstrates how to run one of the
sample applications, SparkPi, packaged with Spark. It
computes an approximation to the value of pi.
| Option | Description |
|---|---|
| application jar | Path to a JAR file containing a Spark application. For the client deployment mode, the path must point to a local file. For the cluster deployment mode, the path can be either a local file or a URL globally visible inside your cluster; see Advanced Dependency Management. |
| python file | Path to a Python file containing a Spark application. For the client deployment mode, the path must point to a local file. For the cluster deployment mode, the path can be either a local file or a URL globally visible inside your cluster; see Advanced Dependency Management. |
| application arguments | Arguments to pass to the main method of your application. |
