Upgrading Apache Spark

The purpose of this documentation is to gather all information required to upgrade Apache Spark to a higher version in Cloudera on cloud:

• upgrading from Spark 2 to Spark 3 (when upgrading from Cloudera on cloud 7.2.18).
• uprading Spark 3.4 to Spark 3.5.4 (when upgrading from Cloudera on cloud 7.3.1.0 to 7.3.1.100 or 7.3.2 or higher).

The necessary set of steps largely depends on the source and target Spark versions, while major version changes require considerable effort, minor and maintenance version changes mostly require only small config or no adjustments.

These guides cover both major upgrades (from Spark 2 to Spark 3) and Spark 3 minor-line upgrades (for example toward Apache Spark 3.5.4).

Major version migration

Migration between major versions requires considerable effort and taking into account many factors.