Upgrading Apache Spark

The purpose of this documentation is to gather all information required to upgrade Apache Spark on CDP Private Cloud 7.1.9:

• upgrading from Spark 2.4.8 to Spark 3.3.x by installing and activating the CDS 3.3 parcel.
• upgrading from Spark 3.3.x to Spark 3.5.4 by installing and activating the CDS 3.5 parcel.

The necessary set of steps largely depends on the source and target Spark versions, while major version changes require considerable effort, minor and maintenance version changes mostly require only small config or no adjustments.

These guides cover both major upgrades (from Spark 2 to Spark 3) and Spark 3 minor-line upgrades (from Spark 3.3.x to Spark 3.5.4). For Spark 3.5.4-specific behavior notes, see the topics under Spark behavior changes in this guide.

Major version migration

Migration between major versions requires considerable effort and taking into account many factors.