Upgrading Apache Spark
Upgrading Spark 2 to Spark 3 for Cloudera Private Cloud 7.3.1
Upgrading from 7.1.9 SP1
Upgrade from Spark 2.4.8
Pre-application migration tasks
Spark application migration (from Spark 2 to Spark 3)
Post-application migration tasks
In-place cluster upgrade
Spark application migration (from Spark 3.x to Spark 3.4.1)
Final steps
Upgrade from Spark 2.4.8 (with CDS 3.3.2)
Application migration tasks from Spark 2 to Spark 3
Post-application migration tasks
In-place cluster upgrade
Spark application migration (from Spark 3.x to Spark 3.4.1)
Final steps
Upgrade from Spark 3.3.2 (CDS)
In-place cluster upgrade
Spark application migration (from Spark 3.x to Spark 3.4.1)
Final steps
Upgrading from 7.1.8
Upgrade from Spark 2.4.8
Pre-application migration tasks
Application migration tasks (Spark 2 to Spark 3)
Post-application migration tasks
In-place cluster upgrade
Application migration tasks (Spark 3.x to Spark 3.4.1)
Final steps
Upgrade from Spark 2.4.8 (with connectors)
Intermediate in-place cluster upgrade
Upgrade from Spark 2.4.8 (with CDS 3.3.x)
Application migration tasks (from Spark 2 to 3)
Post-application migration tasks
In-place cluster upgrade
Application migration tasks (from Spark 3.x to 3.4.1)
Final steps
Upgrade from Spark 2.4.8 (with CDS 3.3.x and connectors)
Intermediate in-place cluster upgrade
Upgrade from Spark 3.3.x (CDS)
In-place cluster upgrade
Application migration tasks (Spark 3.x to Spark 3.4.1)
Final steps
Upgrading from 7.1.7
Upgrade from Spark 2.4.7
Pre-application migration tasks
Application migration tasks (Spark 2 to 3)
Post-application migration tasks
In-place cluster upgrade
Application migration tasks (Spark 3.x to 3.4.1)
Fianl steps
Upgrade from Spark 2.4.7 (with connectors)
Intermediate in-place cluster upgrade
Upgrade from Spark 2.4.7 (with CDS 3.2.3)
Application migration tasks (Spark 2 to 3)
Post-application migration tasks
In-place cluster upgrade
Application migration tasks (Spark 3.x to 3.4.1)
Final steps
Upgrade from Spark 2.4.7 (with CDS 3.2.3 and connectors)
Intermediate in-place cluster upgrade
Upgrade from Spark 3.2.3 (CDS)
In-place cluster upgrade
Application migration tasks (Spark 3.x to 3.4.1)
Final steps
Migrating Spark applications
Java versions
Scala versions
Python versions
Spark commands
Spark connectors
Logging
Third-party libraries
Spark behavior changes
Apache Spark Migration guides
Spark 2 to Spark 3 workload refactoring
Unsupported features
Post-migration checklist
Benchmark testing
Troubleshooting