Handling upgrade failures for Cloudera Data Engineering

If your upgrade of Cloudera Data Engineering (CDE) fails, you have the option to clones the service with the latest version of CDE. Learn how to handle an upgrade failure.

During the Upgrading Cloudera Data Engineering process, linked below, a backup is created as part of the upgrade preparation process. This procedure will use that backup to be restored in a new cluster. The list of service backups is available in the Backup Library. To locate the Backup Library, in CDE, click Administration in the left navigation menu, click Service Details, click the Maintenance tab, and click Backup Library. Also, the list of all available backups can be obtained with the cdp de list-backups CDP CLI command. The list of backups of services that were associated with a specific CDP environment can be obtained with the "cdp de list-backups --filter "environment(eq)<cdp-env-name>" command, where <cdp-env-name> is the name of CDP environment. The list of service backups are available in the Backup Library in the CDE UI by going to Administration > Service Details > Maintenance > Backup Library.

The CDE backup includes the following:

  • CDE Service configurations
  • Virtual cluster names
  • Virtual cluster configurations
  • Virtual cluster file-based resources
  • Spark job definitions
  • Airflow job definitions
  • Spark Python-env resources
The following are not yet included in the backup:
  • Non file-based resources, for example, Python-venv resources and custom runtimes
  • Airflow custom operators & libraries
  • Logs
  • Job run history
  • End points


1. You must re-name and assign a new ID by using the following options:
--service-id and --service-name
  1. Restore the service from the backup. Note that backup-id is the ID of the backup that you are restoring from, and environment-crn is the Customer Resource Number (CRN) of the Cloudera Data Platform (CDP) environment with which a restored CDE service will be associated. Currently, the CDE service can be restored only to the same CDP environment where a backed-up service was associated:
    cdp de restore-service --backup-id <backup-id> --environment-crn
    For example:
    cdp de restore-service --backup-id 2 --environment-crn crn:cdp:environments:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:environment:c67b9089-2d3b-4579-861d-c0df12a105b1
  2. Optional: To obtain a list of backups, use the following command:
    cdp de list-backups
    1. You can describe a particular backup with the following command:cdp de describe-backup --backup-id <backup-id>
      For example:
      $ cdp de describe-backup --backup-id 2 --profile priv
          "backup": {
              "id": 2,
              "serviceID": "cluster-cf6h74lq",
              "serviceName": "dex-priv-default-azure-env-1689008683873",
              "environmentName": "dex-priv-default-azure-env",
              "environmentCrn": "crn:cdp:environments:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:environment:c67b9089-2d3b-4579-861d-c0df12a105b1",
              "creator": "crn:altus:iam:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:user:0f9a97a7-23a7-43bd-bc71-ecdb2aa34ed5",
              "cloudPlatform": "AZURE",
              "status": "completed",
              "created": "2023-07-17T18:02:58.385455Z"