Upgrading Apache Spark 2 to Spark 3
Upgrading Apache Spark from version 2 to version 3 in Cloudera Cloudera on premises is a process that involves:
- Intermediate in-place cluster upgrade tasks, due to different support for Spark 3 versions of connectors in Cloudera versions.
- Intermediate Spark application migration tasks, due to minor or maintenance Spark version changes.
- Pre-application migration tasks.
- Application migration from Spark 2 to Spark 3, due to major Spark version changes.
- Post-application migration tasks.
In-place cluster upgrade tasks.
- Spark application migration tasks, due to minor or maintenance Spark version changes.
This table below provides links to the upgrade guides based on the version of Cloudera on premises you're using, and the source versions of Spark in your environment.
Each upgrade guide contain all steps needed to upgrade Spark 2 to Spark 3 and perform the upgrade to Cloudera on premises version 7.3.1.
| Source cluster version | Source cluster Spark 2 version | Source cluster Spark 3 version | Documentation |
|---|---|---|---|
| 7.1.9 SP1 | 2.4.8 | None | Upgrade guide |
| 2.4.8 | 3.3.2 (CDS parcel) | Upgrade guide | |
| None | 3.3.2 (CDS parcel) | Upgrade guide | |
| 7.1.8 | 2.4.8 | None | Upgrade guide |
| Upgrade (with connectors) guide | |||
| 2.4.8 | 3.3.0 or 3.3.2 (CDS parcel) | Upgrade guide | |
| Upgrade guide (with connectors) | |||
| None | 3.3.0 or 3.3.2 (CDS parcel) | Upgrade guide | |
| 7.1.7 SP1 | 2.4.7 | None | Upgrade guide |
| Upgrade guide (with connectors) | |||
| 2.4.7 | 3.2.3 (CDS parcel) | Upgrade guide | |
| Upgrade guide (with connectors) | |||
| None | 3.2.3 (CDS parcel) | Upgrade guide |
Spark 3 minor upgrade to Apache Spark 3.5.4
Apache Spark 3.5.4 is included in Cloudera Runtime 7.3.1.100 and higher, and 7.3.2.0. Use these guides when you already run Spark 3 and need to move to Spark 3.5.4.
| Source Spark 3 version | Target Cloudera Runtime (Spark 3.5.4) | Documentation |
|---|---|---|
| 3.4.1 (bundled) | 7.3.1.100 and higher, or 7.3.2.0 | Upgrade guide |
| Source cluster version | Source cluster Spark 3 version | Target Cloudera Runtime (Spark 3.5.4) | Documentation |
|---|---|---|---|
| 7.1.9 SP1 | 3.3.2 (CDS parcel) | 7.3.1.100 and higher, or 7.3.2.0 | Upgrade guide |
