Running Apache Spark 3.5.4 applications

spark-shell on JDK 17

On clusters that run JDK 17, the Java Platform Module System (JPMS) enforces strong encapsulation on java.base. The interactive Scala REPL used by spark-shell must resolve Java base classes through the application classpath. If the driver JVM does not include -Dscala.usejavacp=true, spark-shell initialization can fail with errors such as:

[init] error: error while loading Object, Missing dependency 'object scala.native in compiler mirror', required by /modules/java.base/java/lang/Object.class
Failed to initialize compiler: object scala in compiler mirror not found.

Pre-compiled jobs submitted with spark-submit and interactive pyspark sessions are not affected.

In Cloudera Manager:

1. Navigate to Clusters > Spark 3 > Configuration.
2. Search for Spark 3 Client Advanced Configuration Snippet (Safety Valve) for spark3-conf/spark-defaults.conf (or spark-defaults.conf).
3. Append -Dscala.usejavacp=true to spark.driver.extraJavaOptions. For example:

   spark.driver.extraJavaOptions=-Dscala.usejavacp=true --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.security=ALL-UNNAMED --add-opens=java.base/sun.reflect.annotation=ALL-UNNAMED

4. Click Save Changes.
5. Redeploy client configuration for Spark 3 (Stale Configuration → Deploy Client Configuration).

Unsupported connectors

Limitations of Spark in Cloudera

Canary test for pyspark command

The following example shows a simple pyspark session that refers to the SparkContext, calls the collect() function which runs a Spark 3 job, and writes data to HDFS. This sequence of operations helps to check if there are obvious configuration issues that prevent Spark 3 jobs from working at all. For the HDFS path for the output directory, substitute a path that exists on your own system.

$ hdfs dfs -mkdir /user/jdoe/spark
$ pyspark
...
SparkSession available as 'spark'.
>>> strings = ["one","two","three"]
>>> s2 = sc.parallelize(strings)
>>> s3 = s2.map(lambda word: word.upper())
>>> s3.collect()
['ONE', 'TWO', 'THREE']
>>> s3.saveAsTextFile('hdfs:///user/jdoe/spark/canary_test')
>>> quit()
$ hdfs dfs -ls /user/jdoe/spark
Found 1 items
drwxr-xr-x   - jdoe spark-users  0 2016-08-26 14:41 /user/jdoe/spark/canary_test
$ hdfs dfs -ls /user/jdoe/spark/canary_test
Found 3 items
-rw-r--r--   3 jdoe spark-users  0 2016-08-26 14:41 /user/jdoe/spark/canary_test/_SUCCESS
-rw-r--r--   3 jdoe spark-users  4 2016-08-26 14:41 /user/jdoe/spark/canary_test/part-00000
-rw-r--r--   3 jdoe spark-users 10 2016-08-26 14:41 /user/jdoe/spark/canary_test/part-00001
$ hdfs dfs -cat /user/jdoe/spark/canary_test/part-00000
ONE
$ hdfs dfs -cat /user/jdoe/spark/canary_test/part-00001
TWO
THREE

Fetching Spark 3 Maven Dependencies

The Maven coordinates are a combination of groupId, artifactId and version. The groupId and artifactId are the same as for the upstream Apache Spark project. For example, for spark-core, groupId is org.apache.spark, and artifactId is spark-core_2.12, both the same as the upstream project. The version is different for the Cloudera packaging, it contains the upstream Spark version as well as the Cloudera version: [**artifactId***]-[***UPSTREAM SPARK VERSION***].[***CDP VERSION***]-[***CLOUDERA BUILD NUMBER***]. Use the Cloudera Runtime release notes or Maven repository documentation for your deployment to obtain the exact version string for Spark 3.5.4.

Accessing the Spark 3 History Server

The Spark 3 history server is available on port 18089, rather than port 18088 as with the Spark 2 history server.