Updating Spark 2 applications for Spark 3.4

You must update your Apache Spark 2 applications to run on Spark 3. The Apache Spark documentation provides a migration guide.

For instructions on updating your Spark 2 applications for Spark 3, see the Migration Guide in the Apache Spark documentation.

important

The yarn-client and yarn-cluster values for spark.master were deprecated in Spark 2.x (see SPARK-13220), and removed in Spark 3.0 (see SPARK-28980).

Please use yarn and specify the deploy mode instead. For example:

./bin/spark-submit --class [*** PATH TO YOUR CLASS***] --master yarn --deploy-mode cluster [*** OPTIONS ***] [*** APP JAR***] [*** APP OPTIONS ***]

For more information, refer to the relevant topic in the Spark documentation.

Oozie also supports Spark 3 actions. For more information, see Spark 3 support in Oozie.