Cloudera Docs

Upgrading Spark 3.3.2 (CDS) on 7.1.9 SP1 to Spark 3.5.4

The following steps help you move from Apache Spark 3.3.2 (CDS parcel) on CDP Private Cloud 7.1.9 SP1 to Apache Spark 3.5.4 (CDS 3.5 parcel) on the same cluster version.

Source cluster version Source cluster Spark 3 version Target cluster version Target cluster Spark 3 version Spark 2 used with connectors1
7.1.9 SP1 3.3.2 (CDS parcel) 7.1.9 SP1 3.5.4 (CDS 3.5 parcel) no
1 Oozie, Solr, Phoenix, Hive Warehouse Connector, Spark Schema Registry

Activate CDS 3.5.4 parcel

Install and activate the CDS 3.5 parcel on your cluster, as described in the CDS 3.5 parcel documentation. A short overview of the process is as follows:
  1. Confirm that your cluster meets the CDS 3.5 software requirements.
  2. Stop your existing Spark 3 jobs.
  3. In the Admin Console, add the CDS 3.5 parcel repository to Remote Parcel Repository URLs in Parcel Settings.
  4. Download the CDS 3.5 parcel, distribute it to the hosts in your cluster, deactivate the CDS 3.3 parcel, and activate the CDS 3.5 parcel.
  5. Return to the Home page.
  6. Click the stale configuration icon to launch the Stale Configuration wizard and restart the necessary services.
  7. After you confirm that your applications are working on Spark 3.5.4, you can remove the CDS 3.3 parcel from your cluster. See Managing parcels for more information.

Spark application migration

Follow the Spark application migration documentation to migrate your Apache Spark applications from version 3.3.2 to 3.5.4.
  1. Review behavior changes for Spark 3.3.x through 3.5.4 and refactor your Spark application code.

Final steps

After the upgrade and application migration are complete:
  1. Check the status of your clusters.
  2. Perform benchmark testing on your applications. See Benchmark testing.