Upgrading Cloudera Data Engineering

Cloudera Data Engineering (CDE) supports in-place upgrades on both AWS and Azure. CDE provides an in-place upgrade option from the eligible versions to the latest version. If you are on a CDE version that is not eligible for the in-place upgrade, the in-place upgrade option is disabled and you can perform a backup and restore. The upgrades can be triggered by an Admin from the CDE user interface.

To check whether your CDE version is eligible for the in-place upgrade option, see Support lifecycle policy.

When you upgrade to the latest version in Cloudera Data Engineering, the upgrade process prepares your service for the upgrade with a click of a button. Your information is also backed up in this process. Learn how to complete the upgrade process here.

important

Virtual clusters for this service are unavailable during the upgrade process and the upgrade puts your service into maintenance. During this time, scheduled jobs are paused and any active jobs are killed. The upgrade can take some time so be advised and alert your team of the downtime.
Before the upgrade, delete any existing Airflow operators and libraries. If you have configured and built any Airflow libraries and operators, activate and then delete them before the upgrade. For more information, see In-place upgrade with Airflow Operators and Libraries and Upgrading Airflow if DAGs and packages are compatible with new Airflow version.

The following are not included in the backup:

Python-venv resources
Airflow custom operators and libraries
Airflow connections
Airflow variables
Logs
- Job Run logs (Driver, Executor, API)
- Service logs
- Virtual Cluster event logs
- Spark Session logs (Including statement history)
- Airflow DAG logs
Job Runs
- Job Run history
Virtual Cluster end-points
Resource: Docker runtimes

Ensure that the catchup option is not enabled for any user's Airflow jobs. Before the backup starts, if the Airflow DAG catchup options are enabled, disable them manually.

In the Cloudera Data Platform (CDP) console, click the Data Engineering tile.
The CDE Home page displays.
In the left navigation menu, click Administration, select a Service, and click Upgrade.
Click the Maintenance tab and click Upgrade Service.
Click Start Preparation.
CDE prepares the service for you. If the service cannot be prepared, click Retry or Cancel Upgrade. Canceling the upgrade returns your service to the original state. Optionally, you can click Download Logs to view the Diagnostics page where you can generate a diagnostic bundle. If the preparation is successful, proceed to the next step. If the preparation and cancellation fails, you must contact Cloudera support.
Click Start Backup.

CDE automatically creates a backup for you. The backup is used for restoring if an irrecoverable failure happens during the upgrade.

note
If the backup fails, you have the option to click Retry Upgrade or Contact Support. If the upgrade fails a second time, you have the option to contact support or perform a manual upgrade with jobs backup and restore. If the backup is successful, you can proceed with the upgrade.
Optional: When the backup is complete, to view the backup that was created, select Backup Library.
Click Start Upgrade.
After the upgrade is complete, to enable all paused jobs and sessions, click Resume Service.
The service restarts.

After the upgrade is complete and you resumed the service, the service operates on the upgraded version of CDE. If you cancel the upgrade, your service resumes the previous state, which enables all paused jobs and resumes all sessions.