Backing up CDE service using the docker image

You must run the docker image to take a backup of a Cloudera Data Engineering (CDE) service. It takes backup of all the active virtual clusters in that CDE service. You can take backup of only one active CDE service at a time.

You must download the dex-upgrade-utils docker image and create the cde-upgrade-util.properties file before backing up jobs as described in the Prerequisites for upgrading CDE Service with endpoint stability section.

Set the following environment variables in the ECS Server host terminal:

export PATH=$PATH:/opt/cloudera/parcels/ECS/installer/install/bin/linux/:/opt/cloudera/parcels/ECS/docker
export KUBECONFIG=~/kubeconfig
export BASE_WORK_DIR=/opt/backup-restore
export BACKUP_OUTPUT_DIR=/home/dex/backup

Run the dex-upgrade-utils docker image on the host machine:

docker run \
-v [***KUBECONFIG_FILE_PATH***]:/home/dex/.kube/config:ro \
-v [***CDP_CREDENTIAL_FILE_PATH***]:/home/dex/.cdp/credentials:ro \
-v [***CDE-UPGRADE-UTIL.PROPERTIES_FILE_PATH***]:/opt/cde-backup-restore/scripts/backup-restore/cde-upgrade-util.properties:ro \
-v [***LOCAL_BACKUP_DIRECTORY***]:$BACKUP_OUTPUT_DIR \
-e KUBECONFIG=/home/dex/.kube/config \
[***DOCKER_IMAGE_NAME***]:[***DOCKER_IMAGE_VERSION***] prepare-for-upgrade -s [***CDE-CLUSTER-ID***] -o $BACKUP_OUTPUT_DIR

Example:

docker run \

-v $BASE_WORK_DIR/secrets/kubeconfig:/home/dex/.kube/config:ro \
-v $BASE_WORK_DIR/secrets/credentials:/home/dex/.cdp/credentials:ro \
-v $BASE_WORK_DIR/cde-upgrade-util.properties:/opt/cde-backup-restore/scripts/backup-restore/cde-upgrade-util.properties:ro \
-v $BASE_WORK_DIR/backup:$BACKUP_OUTPUT_DIR \
-e KUBECONFIG=/home/dex/.kube/config \
docker-private.infra.cloudera.com/cloudera/dex/dex-upgrade-utils:1.20.1 prepare-for-upgrade -s cluster-c2dhkp22 -o $BACKUP_OUTPUT_DIR

You have now taken the Cloudera Data Engineering (CDE) service backup as a ZIP file. You can make a note of the Zip file name from the logs to use it while restoring the CDE service.

You must now expand the resource pool, and then upgrade your Cloudera Data Platform (CDP) before you restore the CDE service. For information about configuring resource pool and capacity, see Managing cluster resources using Quota Management.

important

During the restore operation, both old and the new CDE services use the same resources allocated to the existing CDE service. Hence, you must double the resource pool size using the Quota Management option. For example, if root.default.sales is the pool that is used for the old or existing CDE service, you must double the CPU and memory resources of this pool. Also, make sure that you have sufficient hardware when doubling the resource pool size. Consider the following conditions and plan whether to modify the resource pool size or not:

If the CDE service uses the default resource pool, that is root.default, then do not change the resource pool size.
If the CDE service uses a custom resource pool (for example, root.default.primary.secondary), the resource pool size of the last level (that is, secondary level in the example) must be doubled using the Quota Management option. The additional capacity required after doubling the last level's pool size is allocated from the levels above it, starting from the higher levels and progressing downward. In this example, when you double the secondary level (last level), the extra resource pool capacity required is initially added to the primary level pool. Then the newly added resource pool capacity is added to the secondary level pool, resulting in an overall doubling of the resource pool size of the last level.
The resource capacity at the CDE service and the Virtual Cluster level must not be changed. Modifying the pool size at the resource pool level is sufficient.