Using the backup-restore-based upgrade script

Learn about how to use the backup-restore-based upgrade script in Cloudera Data Engineering on cloud.

This procedure applies to Cloudera Data Engineering versions 1.20.3-h2, 1.21.0-h2, 1.22.0, and higher.

If Apache Airflow connections and variables are involved, the original backup-restore-based upgrade is not applicable. This document describes a script-based fallback for performing the backup-restore-based upgrade that you can use when the default upgrade method described in Handling upgrade failures for Cloudera Data Engineering does not work.

Table 1. Contents included in the backup
Artifacts	Included in the automation script
Service config	Y
Service Logs	N
Virtual Cluster config	Y
Virtual Cluster end-points	N
Virtual Cluster event logs	N
Job: Spark	Y
Job: Airflow	Y
Resource: files	Y
Resource: docker runtimes	Y
Resource: python-venv (for Spark)	Y
Resource: python-venv (for Airflow)	Y (Since 1.21.0)
Git Repository	Y
Credentials	Y
Spark Session	N
Spark Session logs (Including statement history)	N
Job Runs	Y (Since 1.22.0)
Job Run logs (Driver, Executor, API)	N
Airflow DAG logs	N
Airflow connections	Y
Airflow variables	Y

Once the backup is complete, Service and Virtual Cluster metadata are stored in the Object Store. Data related to jobs and resources is stored locally, at the location where you invoked the backup command.

Before running the script, you must install the listed tools:

CDP CLI
cdpcurl
CDE CLI
jq
kubectl

Install CDP CLI. For more information, see Installing Cloudera client.
Install cdpcurl. For more information, see cdpcurl.
Download the CDE CLI from your Virtual Cluster page and add its path to the PATH environment variable with the export PATH="$PATH:[***CLI-PATH***]" command.
Install the other tools according to your system requirements.

Configure the CDP CLI.
1. On the Cloudera Management Console, navigate to your Profile page.
2. On the Access Keys tab, click Generate Access Key.
3. Download the credentials file and copy the contents to ~/.cdp/credentials
  The credentials are set as the default profile. You can also rename the profile. If you do so, set the CDP profile to your preference using one of the listed methods:
  - The CDP_PROFILE environment variable
  - The cde-service-backup-restore-utils.properties file
  - The --cdp-profile command line option
Configure the CDE CLI.
Configure the CDE CLI following the instructions in Configuring the CLI client. Cloudera recommends that you use the Cloudera credentials, as they are needed for CDP CLI too.
1. Set the CDE CLI using one of the listed methods:
  - The environment variable
  - The cde-service-backup-restore-utils.properties file
  - The ~/.cde/config.yaml file
  note
  Command line options are not available.
2. Run the following command once to make sure that the configuration works:
  cde job list --vcluster-endpoint [***JOBS-API-URL***]
Back up the Cluster.
1. Download the backup-restore-based upgrade script (cde-service-backup-restore-utils.sh) to your local machine: navigate to https://github.com/cloudera/cde-public/blob/master/scripts/cde-service-backup-restore-utils.sh and download the raw file using the download icon on the right.
2. Authenticate to kubectl to access your Cluster.
3. To back up the Cluster, run the following command.
  note
  Using --pre-check is optional, but recommended on the first attempt. With --pre-check, you can check the output of the tools in advance, to make sure the tools are properly installed and configured.
```
./cde-service-backup-restore-utils.sh backup-service -s <cluster-id> -b <backup-base-directory> --pre-check
```
Some of the contents are stored remotely and some are stored locally in the directory you specify.
note
-b indicates the backup base directory, which is optional. If not specified, the default $HOME/.cde/service-backups value is used. Backup files are saved under [***BASE-DIR***]/[***SERVICE-ID***]/, which means that the same directory is shared between multiple backups with the same base directory and service ID. If you want to preserve multiple backups of the same service, specify a different base directory. When you are sure that the backup is no longer needed, delete the backup files.
Restore the Cluster.
1. Run the following command to provision the Cluster and Virtual Clusters. This can take around an hour.
  note
  To retain the endpoints of a VC, obtain the original service ID, after that, make sure that you delete the original Cloudera Data Engineering service. To obtain the service ID, on the Cloudera Data Engineering UI, select Administration in the left navigation menu. Select the service and check the ID on the Service Details page.
  To retain the endpoints of a VC, include the --service-id [***ORIGINAL-SERVICE-ID***] parameter in the restore-service CLI command.
  If you do not specify a cluster ID in the restore-service command, a random ID is generated.
```
./cde-service-backup-restore-utils.sh restore-service -s <original-cluster-id> -t <new-cluster-id> -n <new-cluster-name> -b <backup-base-directory> --service-only
```
2. Authenticate again to kubectl to access your Cluster.
  If the version of the new service is different from the original one, download the CDE CLI tool of the new service and add its path to the PATH environment variable.
  note
  
  Ideally, all scheduled jobs are paused before taking the backup. If the scheduled jobs were not paused, add the --pause-jobs flag to the command to pause them on restore. If you add this flag, the script modifies the Airflow DAGs also to stop catch up.
  
  To restore your contents, run:
```
./cde-service-backup-restore-utils.sh restore-service -s <original-cluster-id> -t <new-cluster-id> -n <new-cluster-name> -b <backup-base-directory> --contents-only
```