Upgrading Cloudera Data Engineering

Cloudera Data Engineering (CDE) supports in-place upgrades on both AWS and Azure. You can upgrade your service from CDE 1.19.2 and above for AWS and from CDE 1.19.4 and above for Azure to the latest CDE release. The upgrades can be triggered by an Admin from the CDE user interface.

When you upgrade to the latest version in Cloudera Data Engineering, the upgrade process will prepare your service for the upgrade with a click of a button. Your information will also be backed up for you in this process. Learn how to complete the upgrade process here.

important

In-place upgrades GA do not work with AWS and Azure private clusters which are in tech preview. Instead of using an in-place upgrade for AWS and Azure private clusters, you must upgrade to the latest CDE version by cloning your CDE service and copying over jobs using the jobs backup & restore function.
Virtual clusters for this service are unavailable during the upgrade process and the upgrade will put your service into maintenance. During this time, scheduled jobs are paused and any active jobs are killed. The upgrade can take some time so be advised and alert your team of the downtime.

The following are not included in the backup:

Python-venv resources
Airflow custom operators & libraries
Logs
Job run history

Ensure that the option catchup is not enabled for any user's Airflow jobs. You must manually disable the Airflow DAG catchup options if they are enabled before the backup starts.

In the Cloudera Data Platform (CDP) console, click the Data Engineering tile.
The CDE Home page displays.
Click Administration in the left navigation menu, select a Service, and click Upgrade.
Click the Maintenance tab and click Upgrade Service.
Click Start Preparation. CDE prepares the service for you. If the service can not be prepared, you can click Retry or Cancel Upgrade. Canceling the upgrade returns your service to the original state. Optionally, you can click Download Logs to view the Diagnostics page where you can generate a diagnostic bundle. If the preparation is successful, proceed to the next step. If the preparation and cancellation fails, you must contact Cloudera support.
Click Start Backup. CDE automatically creates a backup for you. Optionally, you can click Download Logs to view the Diagnostics page where you can generate a diagnostic bundle.

note
If the backup fails, you have the option to click Retry Upgrade or Contact Support. If the upgrade fails a second time, you have the option to contact support or run a side-by-side upgrade. If the backup is successful, you can proceed with the upgrade.
Optional: When the backup is complete, you can click Backup Library to view the backup that was created.
Click Start Upgrade.
Once the upgrade is complete, click Resume Service to enable all paused jobs and sessions.
The service restarts.

Once the upgrade is complete, and you have resumed service, the service will operate on the upgraded version of CDE. If you wish to cancel the upgrade, your service will resume to the previous state which will enable all paused jobs and resume all sessions.