Cloudera Docs

Upgrading Apache Spark 2.4.8 (with CDS 3.3.x) on 7.1.8 to Spark 3 on 7.3.1

The following steps will help you upgrading from Apache Spark 2.4.8 (with CDS 3.3.x) on Cloudera Private Cloud 7.1.8 to Spark 3.4.1 on 7.3.1.

Source cluster version Source cluster Spark 2 version Source cluster Spark 3 version Target cluster version Target cluster Spark 3 version Spark 2 used with connectors1
7.1.8 2.4.8 3.3.0 or 3.3.2 (CDS) 7.3.1 3.4.1 yes

Intermediate in-place cluster upgrade

  1. Upgrade your cluster to Cloudera Private Cloud 7.1.9 SP1 with CDS 3.3.2.
    1. Review the Supported Upgrade Paths for your upgrade.
    2. Review the Requirements and Supported Versions for your upgrade
    3. Review the Release Notes for the version of Cloudera Private Cloud Base you are upgrading to.
    4. Review the Cloudera Upgrade/Migrate Troubleshooting Articles.
    5. Gather information on your deployment.
    6. Plan how and when to begin your upgrade.
    7. If necessary, Upgrade the JDK.
    8. If necessary, Upgrade the Operating System.
    9. Upgrade Cloudera Manager.
    10. Use Cloudera Manager to upgrade from lower version of Cloudera Runtime to a higher version of Cloudera Runtime.
    11. Check that all the software prerequisites are satisfied for CDS 3.3.2.
    12. In the Cloudera Manager Admin Console, add the CDS parcel repository to the Remote Parcel Repository URLs in Parcel Settings.
    13. Download the CDS parcel, distribute it to the hosts in your cluster, and activate it.
    14. Add the Spark 3 service to your cluster.
    15. Return to the Home page.
    16. Click the stale configuration icon to launch the Stale Configuration wizard and restart the necessary services.

    For more information on in-place cluster upgrades and installing the CDS parcel, see:

  2. Once complete, continue the upgrade with the steps described in Upgrading Apache Spark 2 (with 3.3.2 CDS) on 7.1.9 SP1 to Spark 3 on 7.3.1.
1 Oozie, Solr, Phoenix, Hive Warehouse Connector, Spark Schema Registry