Upgrading Spark 2 to Spark 3 for Cloudera on premises 7.3.1
Upgrading Apache Spark from version 2 to version 3 in Cloudera Cloudera on premises is a process that involves:
- Intermediate in-place cluster upgrade tasks, due to different support for Spark 3 versions of connectors in Cloudera versions.
- Intermediate Spark application migration tasks, due to minor or maintenance Spark version changes.
- Pre-application migration tasks.
- Application migration from Spark 2 to Spark 3, due to major Spark version changes.
- Post-application migration tasks.
In-place cluster upgrade tasks.
- Spark application migration tasks, due to minor or maintenance Spark version changes.
This table below provides links to the upgrade guides based on the version of Cloudera on premises you're using, and the source versions of Spark in your environment.
Each upgrade guide contain all steps needed to upgrade Spark 2 to Spark 3 and perform the upgrade to Cloudera on premises version 7.3.1.
| Source cluster version | Source cluster Spark 2 version | Source cluster Spark 3 version | Documentation |
|---|---|---|---|
| 7.1.9 SP1 | 2.4.8 | None | Upgrade guide |
| 2.4.8 | 3.3.2 (CDS parcel) | Upgrade guide | |
| None | 3.3.2 (CDS parcel) | Upgrade guide | |
| 7.1.8 | 2.4.8 | None | Upgrade guide |
| Upgrade (with connectors) guide | |||
| 2.4.8 | 3.3.0 or 3.3.2 (CDS parcel) | Upgrade guide | |
| Upgrade guide (with connectors) | |||
| None | 3.3.0 or 3.3.2 (CDS parcel) | Upgrade guide | |
| 7.1.7 SP1 | 2.4.7 | None | Upgrade guide |
| Upgrade guide (with connectors) | |||
| 2.4.7 | 3.2.3 (CDS parcel) | Upgrade guide | |
| Upgrade guide (with connectors) | |||
| None | 3.2.3 (CDS parcel) | Upgrade guide |
