Handling upgrade failures for Cloudera Data Engineering

If your upgrade of Cloudera Data Engineering (CDE) fails, you have the option to clones the service with the latest version of CDE. Learn how to handle an upgrade failure.

During the Upgrading Cloudera Data Engineering process, linked below, a backup is created as part of the upgrade preparation process. This procedure will use that backup to be restored in a new cluster. The list of service backups is available in the Backup Library. To locate the Backup Library, in CDE, click Administration in the left navigation menu, click Service Details, click the Maintenance tab, and click Backup Library. Also, the list of all available backups can be obtained with the cdp de list-backups CDP CLI command. The list of backups of services that were associated with a specific CDP environment can be obtained with the "

cdp de list-backups --filter
                "environment(eq)<cdp-env-name>

" command, where <cdp-env-name> is the name of CDP environment. The list of service backups are available in the Backup Library in the CDE UI by going to Administration > Service Details > Maintenance > Backup Library.

The CDE backup includes the following:

CDE Service configurations
Virtual cluster names
Virtual cluster configurations
Virtual cluster file-based resources
Spark job definitions
Airflow job definitions
Spark Python-env resources

The following are not yet included in the backup:

Non file-based resources, for example, Python-venv resources and custom runtimes
Airflow custom operators & libraries
Logs
Job run history
End points

Steps:

1. You must re-name and assign a new ID by using the following options:

--service-id and --service-name

Restore the service from the backup. Note that backup-id is the ID of the backup that you are restoring from, and environment-crn is the Customer Resource Number (CRN) of the Cloudera Data Platform (CDP) environment with which a restored CDE service will be associated. Currently, the CDE service can be restored only to the same CDP environment where a backed-up service was associated:
```
cdp de restore-service --backup-id <backup-id> --environment-crn
    <environment-crn>
```
For example:
```
cdp de restore-service --backup-id 2 --environment-crn crn:cdp:environments:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:environment:c67b9089-2d3b-4579-861d-c0df12a105b1
```

Optional: To obtain a list of backups, use the following command:

cdp de list-backups

You can describe a particular backup with the following command:

cdp de describe-backup --backup-id
                                <backup-id>

For example:

$ cdp de describe-backup --backup-id 2 --profile priv
{
    "backup": {
        "id": 2,
        "serviceID": "cluster-cf6h74lq",
        "serviceName": "dex-priv-default-azure-env-1689008683873",
        "environmentName": "dex-priv-default-azure-env",
        "environmentCrn": "crn:cdp:environments:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:environment:c67b9089-2d3b-4579-861d-c0df12a105b1",
        "creator": "crn:altus:iam:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:user:0f9a97a7-23a7-43bd-bc71-ecdb2aa34ed5",
        "cloudPlatform": "AZURE",
        "status": "completed",
        "created": "2023-07-17T18:02:58.385455Z"
    }
}