Cloudera Docs

Upgrading Apache Spark 2.4.8 (with CDS 3.3.2) on 7.1.9 SP1 to Spark 3.3.x

The following steps help you complete the migration from Apache Spark 2.4.8 to Spark 3.3.x on CDP Private Cloud 7.1.9 SP1 when the CDS 3.3.2 parcel is already installed.

Source cluster version Source cluster Spark 2 version Source cluster Spark 3 version Target cluster version Target cluster Spark 3 version Spark 2 used with connectors1
7.1.9 SP1 2.4.8 3.3.2 (CDS) 7.1.9 SP1 3.3.2 (CDS parcel) no
1 Oozie, Solr, Phoenix, Hive Warehouse Connector, Spark Schema Registry

Application migration tasks from Spark 2 to Spark 3

Follow the Spark application migration documentation to migrate your Apache Spark applications from version 2.4.8 to 3.3.2.
  1. Check the supported Java versions.
  2. Check the supported Scala version.
  3. Check the supported Python versions.
  4. Account for changed or versioned Spark commands in your code. (spark-submit, pyspark, etc.)
  5. Check supported versions for Spark connectors.
  6. Check the logging library used in your code.
  7. Check the compatibility of 3rd-party libraries used in your code.
  8. Check Spark behavior changes and refactor your code.

Post-application migration tasks

  1. Stop the Livy (Livy for Spark 2) and Spark 2 (SPARK_ON_YARN) services.
  2. Delete the Spark 2 and Livy for Spark 2 services.
  3. Move Spark 2 event logs to the Spark 3 event logs directory.
  4. Continue after Spark 2 services and roles are fully removed from the cluster.

Final steps

After the upgrade and application migration are complete:
  1. Check the status of your clusters.
  2. Perform benchmark testing on your applications. See Benchmark testing.