Cloudera Docs

Upgrading Spark 3.3.2 (CDS) on 7.1.9 SP1 to Spark 3.5.4

The following steps help you move from Apache Spark 3.3.2 (CDS parcel) on Cloudera on premises 7.1.9 SP1 to Apache Spark 3.5.4, in line with Cloudera Runtime 7.3.1.100 and higher or 7.3.2.0.

Source cluster version Source cluster Spark 2 version Source cluster Spark 3 version Target cluster version Target cluster Spark 3 version Spark 2 used with connectors1
7.1.9 SP1 none 3.3.2 (CDS parcel) 7.3.1.100+ or 7.3.2.0 3.5.4 (CDS parcel) no
1 Oozie, Solr, Phoenix, Hive Warehouse Connector, Spark Schema Registry

Activate CDS 3.5.4 parcel

  1. Confirm prerequisites and supported versions for CDS powered by Apache Spark 3.5.4 on your Cloudera on premises release.
  2. In the Cloudera Manager Admin Console, add the CDS parcel repository to Remote Parcel Repository URLs in Parcel Settings.
  3. Download the CDS 3.5.4 parcel, distribute it to the hosts in your cluster, and activate it.
  4. Redeploy client configuration and restart Spark and related services as prompted.

    For more information, see CDS parcel documentation.

In-place cluster upgrade

  1. Upgrade the Server to 7.13.1.0
    1. Back up the server databases, working directories, and several other entities. These backups can be used to restore your deployment if there are problems during the upgrade.
    2. Upgrade the server software on the host using package commands from the command line (for example, yum on RHEL systems). automates much of this process and is recommend for upgrading and managing your CDH/ clusters.
    3. Upgrade the agent software on all cluster hosts. The upgrade wizard can upgrade the agent software (and, optionally, the JDK), or you can install the agent and JDK software manually. The CDH or software is not upgraded during this process.
    For more information, see Upgrading 7.
  2. Use to upgrade your clusters from a lower version of to the target version for your Spark upgrade (for example, 7.3.1).
    For more information, see Upgrading to a higher version.
  3. handles deactivation of the SPARK3 parcel. The parcel itself is not removed but deactivated.

Spark application migration

Follow the Spark application migration documentation to migrate your Apache Spark applications from version 3.3.2 to 3.5.4.
  1. Review behavior changes for Spark 3.3.x through 3.5.4 and refactor your Spark application code.

Final steps

After the upgrade and application migration are complete:
  1. Check the status of your clusters.
  2. Perform benchmark testing on your applications. See Spark Application Migration.