Pre-upgrade - Upgrading Cloudera Data Engineering service

You can seamlessly upgrade a previous Cloudera Data Engineering service version to a new version.

important

Upgrading Cloudera Data Engineering service from version 1.5.4 SP2 or earlier to 1.5.5 or higher does not support endpoint stability. This means the links to your Cloudera Data Engineering Service and Virtual Cluster will change after the upgrade.
After upgrading Cloudera Data Engineering from version 1.5.4 SP2 or earlier to 1.5.5 or higher, the Cloudera Data Engineering Services and Virtual Clusters that were created in the earlier version does not work in the Cloudera Data Engineering 1.5.5 or higher. Cloudera recommends you not to use the old Cloudera Data Engineering Services and Virtual Clusters that were created before upgrade. Instead, create new Services and Virtual Clusters in the Cloudera Data Engineering 1.5.5 or higher version that you upgraded to and use them.
After upgrading Cloudera Data Engineering, the upgraded Virtual Cluster retains the same base OS images that were used in the source Virtual Cluster before upgrade to ensure maximum compatibility particularly for jobs that depend on particular python and scala versions such as Spark jobs. For example, if an old Virtual Cluster is Redhat insecure based, then the new restored Virtual Cluster will also be Redhat insecure based only and if the old Virtual Cluster is security hardened based, then the new restored Virtual Cluster will also be security hardened based only. No automation path is supported from Redhat insecure to security hardened or vice versa.
Upgrading to Cloudera Data Services on premises 1.5.5 SP2 CHF1 triggers an unsupported, automatic upgrade of Cloudera Data Engineering Virtual Cluster (VC) Spark versions from 3.2.x or 3.3.x to 3.5. Furthermore, backup and restore operations in a Cloudera Data Services on premises environment fail during VC creation if the system attempts to restore an older, incompatible Spark version (such as 3.2.4 or 3.3.2) to the target runtime. To prevent upgrade and restoration failures, especially during Data Lake upgrades from 7.1.9 to 7.3.1.x or higher, do the following:
1. Upgrade your Data Lake to at least 7.1.9 SP1.
2. Create a new VC using Spark 3.5 for each existing 3.2.x or 3.3.x VCs.
3. Migrate the workloads from older VCs to the the new Spark 3.5 VC.
4. Validate the workloads on the new Spark 3.5 VC.
5. Delete the old VC(s) that are still on older Spark versions (for example, Spark 3.3).
6. Proceed with the Data Lake upgrade to 7.3.1 or higher.

Once you upgrade to Cloudera Data Engineering 1.5.5 or higher, the endpoints that you were using in the previous version are not supported. The Cloudera Data Engineering service endpoint migration process lets you migrate your resources, jobs, job run history, Spark jobs’ logs, and event logs from your old cluster to the new cluster.