Running sample Spark applications
You can use the following sample Spark Pi and Spark WordCount sample programs to validate your Spark installation and explore how to run Spark jobs from the command line and Spark shell.
Spark Pi
You can test your Spark installation by running the following compute-intensive example, which calculates pi by “throwing darts” at a circle. The program generates points in the unit square ((0,0) to (1,1)) and counts how many points fall within the unit circle within the square. The result approximates pi.
Follow these steps to run the Spark Pi example:
- Authenticate using
kinit
:kinit <username>
- Run the Apache Spark Pi job in yarn-client mode, using code from
org.apache.spark
:spark-submit --class org.apache.spark.examples.SparkPi \ --master yarn-client \ --num-executors 1 \ --driver-memory 512m \ --executor-memory 512m \ --executor-cores 1 \ examples/jars/spark-examples*.jar 10
Commonly used options include the following:
-
--class
-
The entry point for your application: for example, org.apache.spark.examples.SparkPi.
-
--master
-
The master URL for the cluster: for example,
spark://23.195.26.187:7077
. -
--deploy-mode
-
Whether to deploy your driver on the worker nodes (
cluster
) or locally as an external client (default isclient
). -
--conf
-
Arbitrary Spark configuration property in
key=value
format. For values that contain spaces, enclose“key=value”
in double quotation marks. -
<application-jar>
-
Path to a bundled jar file that contains your application and all dependencies. The URL must be globally visible inside of your cluster: for instance, an hdfs:// path or a file:// path that is present on all nodes.
-
<application-arguments>
-
Arguments passed to the main method of your main class, if any.
Your job should produce output similar to the following. Note the value of pi in the output.
17/03/22 23:21:10 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 1.302805 s Pi is roughly 3.1445191445191445
You can also view job status in a browser by navigating to the YARN ResourceManager Web UI and viewing job history server information. (For more information about checking job status and history, see "Tuning Spark" in this guide.) -