Cloudera Docs

Migrating Spark workloads before a Cloudera Runtime upgrade

Before you upgrade Cloudera Base on premises, migrate your Spark workloads to a compatible Spark version:

Before upgrading Cloudera Base on premises, you must update all Cloudera Data Engineering Virtual Clusters (VCs) to a Spark version supported by your target Cloudera Runtime release. Upgrading without compatible Spark versions will cause the upgrade preparation to fail.
  • Cloudera Runtime 7.3.1 supports Apache Spark version 3.5 or higher.
  • Cloudera Runtime 7.3.2 supports Apache Spark version 3.5.4.

For the Cloudera Runtime versions and the Spark versions that each Cloudera Data Engineering release supports, see Compatibility for Cloudera Data Engineering and Cloudera Runtime components.

  1. In the Cloudera Data Engineering service, create new VCs with the Spark version that the target Cloudera Runtime version supports. For more information about creating VCs, see Creating virtual clusters.
  2. Refactor your current Spark jobs to the supported Spark version. For information on the Spark application migration instructions, see Migrating Spark applications.
  3. Test the migrated Spark jobs by running them.
  4. After ensuring that everything works as expected, delete the old VCs that use lower Spark versions and keep the new VCs.
  5. Upgrade Cloudera Base on premises to the target Cloudera Runtime version by using Cloudera Manager. For more information, see the Cloudera Base on premises upgrade documentation.
  6. Perform the in-place upgrade of your Cloudera Data Engineering service.