Running Sample Spark Applications
You can use the following sample Spark Pi and Spark WordCount sample programs to validate your Spark installation and explore how to run Spark jobs from the command line and Spark shell.
Spark Pi
You can test your Spark installation by running the following compute-intensive example, which calculates pi by “throwing darts” at a circle. The program generates points in the unit square ((0,0) to (1,1)) and counts how many points fall within the unit circle within the square. The result approximates pi.
Follow these steps to run the Spark Pi example:
- Log in as a user with Hadoop Distributed File System (HDFS)
access: for example, your
spark
user, if you defined one, orhdfs
.When the job runs, the library is uploaded into HDFS, so the user running the job needs permission to write to HDFS.
- Navigate to a node with a Spark client and access the
spark2-client
directory:cd /usr/hdp/current/spark2-client
su spark
- Run the Apache Spark Pi job in yarn-client mode, using code from
org.apache.spark:
./bin/spark-submit --class org.apache.spark.examples.SparkPi \ --master yarn-client \ --num-executors 1 \ --driver-memory 512m \ --executor-memory 512m \ --executor-cores 1 \ examples/jars/spark-examples*.jar 10
Commonly used options include the following:
-
--class
-
The entry point for your application: for example, org.apache.spark.examples.SparkPi.
-
--master
-
The master URL for the cluster: for example,
spark://23.195.26.187:7077
. -
--deploy-mode
-
Whether to deploy your driver on the worker nodes (
cluster
) or locally as an external client (default isclient
). -
--conf
-
Arbitrary Spark configuration property in
key=value
format. For values that contain spaces, enclose“key=value”
in double quotation marks. -
<application-jar>
-
Path to a bundled jar file that contains your application and all dependencies. The URL must be globally visible inside of your cluster: for instance, an hdfs:// path or a file:// path that is present on all nodes.
-
<application-arguments>
-
Arguments passed to the main method of your main class, if any.
Your job should produce output similar to the following. Note the value of pi in the output.
17/03/22 23:21:10 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 1.302805 s Pi is roughly 3.1445191445191445
You can also view job status in a browser by navigating to the YARN ResourceManager Web UI and viewing job history server information. (For more information about checking job status and history, see "Tuning Spark" in this guide.) -
WordCount
WordCount is a simple program that counts how often a word occurs in a text file. The
code builds a dataset of (String, Int) pairs called counts
, and saves
the dataset to a file.
The following example submits WordCount code to the Scala shell:
- Select an input file for the Spark WordCount example.
You can use any text file as input.
- Log on as a user with HDFS access: for example, your
spark
user (if you defined one) orhdfs
.The following example uses
log4j.properties
as the input file:cd /usr/hdp/current/spark2-client/
su spark
- Upload the input file to HDFS:
hadoop fs -copyFromLocal /etc/hadoop/conf/log4j.properties /tmp/data
- Run the Spark shell:
./bin/spark-shell --master yarn-client --driver-memory 512m --executor-memory 512m
You should see output similar to the following (with additional status messages):
Spark context Web UI available at http://172.26.236.247:4041 Spark context available as 'sc' (master = yarn, app id = application_1490217230866_0002). Spark session available as 'spark'. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.1.0.2.6.0.0-598 /_/ Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_112) Type in expressions to have them evaluated. Type :help for more information. scala>
- At the
scala>
prompt, submit the job by typing the following commands, replacing node names, file name, and file location with your own values:val file = sc.textFile("/tmp/data") val counts = file.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey(_ + _) counts.saveAsTextFile("/tmp/wordcount")
- Use one of the following approaches to view job output:
-
View output in the Scala shell:
scala> counts.count()
-
View the full output from within the Scala shell:
scala> counts.toArray().foreach(println)
-
View the output using HDFS:
-
Exit the Scala shell.
-
View WordCount job status:
hadoop fs -ls /tmp/wordcount
You should see output similar to the following:
/tmp/wordcount/_SUCCESS /tmp/wordcount/part-00000 /tmp/wordcount/part-00001
-
Use the HDFS
cat
command to list WordCount output:hadoop fs -cat /tmp/wordcount/part-00000
-
-