Post-migration tasks
After the workloads are executed on Spark 2.4, validate the output, and compare the performance of the jobs with CDH/HDP cluster executions.
After the workloads are executed on Spark 2.4, validate the output, and compare the performance of the jobs with CDH/HDP cluster executions. After you perform the post migration configurations, do benchmark testing on Spark 2.4.
Troubleshoot the failed/slow performing workloads by analyzing the application event logs/driver logs and fine tune the workloads for better performance.
-
https://spark.apache.org/docs/2.4.4/sql-migration-guide-upgrade.html
https://spark.apache.org/releases/spark-release-2-4-0.html
https://spark.apache.org/releases/spark-release-2-2-0.html
https://spark.apache.org/releases/spark-release-2-3-0.html
https://spark.apache.org/releases/spark-release-2-1-0.html
https://spark.apache.org/releases/spark-release-2-0-0.html
For additional information about known issues please also refer: