Using Spark 2 from Scala

1. Open Cloudera Data Science Workbench.
2. Select New Project.
3. Enter a Project Name.
4. Choose whether the Project Visibility is Private or Public.
5. Under Initial Setup, choose the Template tab.
6. Select Scala.
7. Click Create Project. Your new project displays sample files.
8. Click Open Workbench.
9. Select Scala engine.
10. Click Launch Session.

The Scala engine typically takes 30 seconds to become ready. This increased startup time is due to the Scala engine automatically creating a Spark context in Apache YARN.

The examples that follow are included under the Scala templates.

Unlike PySpark or Sparklyr, you can access a SparkContext assigned to the spark (SparkSession) and sc (SparkContext) objects on console startup, just as when using the Spark shell. By default, the application name will be set to CDSW_sessionID, where sessionId is the id of the session running your Spark code. To customize this, set the spark.app.name property to the desired application name in a spark-defaults.conf file.

Pi.scala is a classic starting point for calculating Pi using Montecarlo Estimation.

This is the full, annotated code sample.

//Calculate pi with Monte Carlo estimation
import scala.math.random

//make a very large unique set of 1 -> n 
val partitions = 2 
val n = math.min(100000L * partitions, Int.MaxValue).toInt 
val xs = 1 until n 

//split up n into the number of partitions we can use 
val rdd = sc.parallelize(xs, partitions).setName("'N values rdd'")

//generate a random set of points within a 2x2 square
val sample = rdd.map { i =>
  val x = random * 2 - 1
  val y = random * 2 - 1
  (x, y)
}.setName("'Random points rdd'")

//points w/in the square also w/in the center circle of r=1
val inside = sample.filter { case (x, y) => (x * x + y * y < 1) }.setName("'Random points inside circle'")
val count = inside.count()
 
//Area(circle)/Area(square) = inside/n => pi=4*inside/n                        
println("Pi is roughly " + 4.0 * count / n)

Key points to note:

• import scala.math.random

  Importing included packages works just as in the shell, and need only be done once.

• Spark context (sc).
  You can access a SparkContext assigned to the variable sc on console startup.

  val rdd = sc.parallelize(xs, partitions).setName("'N values rdd'")

Since HDFS is the configured filesystem, Spark jobs on Cloudera Data Science Workbench read from HDFS by default.

The file you use must already exist in HDFS. For example, you might load the jedi_wisdom.txt file using the terminal. Click the Terminal link above your Cloudera Data Science Workbench console and enter the following command:

hdfs dfs -put data/jedi_wisdom.txt /tmp

This example reads data from HDFS.

//count lower bound 
val threshold = 2

// this file must already exist in hdfs, add a 
// local version by dropping into the terminal.
val tokenized = sc.textFile("/tmp/data/jedi_wisdom.txt").flatMap(_.split(" "))

// count the occurrence of each word
val wordCounts = tokenized.map((_ , 1)).reduceByKey(_ + _)

// filter out words with fewer than threshold occurrences
val filtered = wordCounts.filter(_._2 >= threshold)

System.out.println(filtered.collect().mkString(","))

Key points to note:

• You add files to HDFS by opening the terminal and running the following command:

  hdfs dfs -put data/jedi_wisdom.txt /tmp

• You access the file from Wordcount scala using the following command:

  sc.textFile("tmp/jedi_wisdom.txt")

Use the following command in the terminal to read text from the local filesystem. The file must exist on all nodes, and the same path for the driver and executors. In this example you are reading the file ebay-xbox.csv.

sc.textFile(“file:///tmp/ebay-xbox.csv”) 

External libraries are handled through line magics. Line magics in the Toree kernel are prefixed with %.