Handling upgrade failures for Cloudera Data Engineering

If your upgrade of Cloudera Data Engineering fails, you have the option to clone the service with the latest version of Cloudera Data Engineering. Learn how to handle an upgrade failure.

During a Cloudera Data Engineering upgrade, a backup is created as part of the upgrade preparation process. The backup comprises all metadata and associated resource files, which are stored in HDFS or Ozone. This procedure uses that backup to be restored in a new cluster.

The list of service backups is available in the Backup Library. To locate the Backup Library, in the left navigation menu of Cloudera Data Engineering select Administration, select Service Details, select the Maintenance tab, and select Backup Library.

To obtain the list of all available backups, use the Backup Library in the Cloudera Data Engineering UI, as described above.

The Cloudera Data Engineering backup contains the following artifacts:


Artifacts	Included in the backup
Service config	Y
Service Logs	N
Virtual Cluster config	Y (except SMTP password)
Virtual Cluster end-points	N
Virtual Cluster event logs	N
Job: Spark	Y
Job: Airflow	Y
Resource: files	Y
Resource: docker runtimes	Y
Resource: python-venv (for Spark)	Y
Resource: python-venv (for Airflow)	Y
Git Repository	Y
Credentials	Y
Spark Session	N
Spark Session logs (including statement history)	N
Job Runs	Y
Job Run logs (Driver, Executor, API)	N
Airflow DAG logs	N
Airflow connections	N
Airflow variables	N
Keytabs	N
Roles of the service and virtual clusters (VCs), that is, access control (ACL) role assignments	N
Certificates (TLS certificates)	N
Data connectors	N

Ensure that the catchup option is not enabled for any user's Airflow jobs.
Before the backup starts, if the Airflow DAG catchup options are enabled, disable them manually.
By default, the restored Cloudera Data Engineering service is assigned the name of the original backed-up service, and a new service ID is generated. To prevent backup failure due to naming conflicts, choose one of the following options:
1. Delete the original service that failed to upgrade during the Cloudera Data Engineering upgrade.
2. Rename the service using the --service-name option.
note
To retain endpoint stability, you can explicitly assign the restored service the ID of the original backed-up service.
Assigning a new ID results in new Fully Qualified Domain Names (FQDNs) for the service and all its virtual clusters, which affects the endpoint stability.
A valid service ID is:
- An 8 character-long alphanumeric string
- Does not contain vowels
- Unique

Restore the service from the backup by using the Backup Library in the Cloudera Data Engineering UI: select the backup that you want to restore from, and start the restore.
Optional: To review the available backups and their details before restoring, use the Backup Library in the Cloudera Data Engineering UI.
Optional: If the email alert is enabled and the SMTP password is set, the password is not restored during the restore process. As a result, you need to manually set the password again.
1. In the Cloudera Data Engineering UI, navigate to Administration in the left navigation pane, and select the required service and then the virtual cluster.
2. Select Virtual Cluster Details > Configuration > Workload Alerts Setup.
3. Enter the SMTP password in the corresponding field and click Apply changes.