Prerequisites for upgrading CDE Service with endpoint stability
You must first download the docker image and create the cde-upgrade-util.properties file to back up CDE services.
- Download the dex-upgrade-utils docker image tarball. The file naming convention is dex-upgrade-utils-<version-number>-<build-number>.tar.gz.
-
Load the downloaded image into the host machine docker runtime:
docker load < dex-upgrade-utils-<version-number>-<build-number>.tar.gzExample:
docker load < dex-upgrade-utils-1.20.1-b20.tar.gz -
Create a directory in the host machine and export that path as
BASE_WORK_DIR.
mkdir <host_machine_path> export BASE_WORK_DIR=<host_machine_path>Example:
mkdir /opt/backup-restore export BASE_WORK_DIR=/opt/backup-restore -
Create backup and secrets directories
in the BASE_WORK_DIR directory and update the access
permissions. The secrets directory stores the
kubeconfig and CDP DE Admin credentials
files. The backup directory will store the
backup file which will be generated when you backup the CDE Service.
cd $BASE_WORK_DIR mkdir backup secrets chmod 775 backup - Place the CDP credentials file of the DEAdmin user and administrator kubeconfig file in the $BASE_WORK_DIR/secrets directory.
-
Create the cde-upgrade-util.properties file with the
information described below and save it in the
$BASE_WORK_DIR directory.
cdp_k8s_namespace:<CDP control plane k8s namespace> cdp_endpoint:<CDP control plane endpoint> credential_file_path:<Path to the DEAdmin user CDP credentials file> de_admin_user:<DEAdmin user-id> de_admin_password:<DEAdmin user's password must be in base64 encoded format. Use the echo -n <password> | base64 command to encode the password. > tls_insecure:<Keep it true if you are using a self-signed certificate> auto_unpause_jobs: <Specify it as “true” if you want to automatically resume the jobs that were paused during the backup phase. The jobs will be resumed after you restore the CDE service.> platform_type:<Platform type, can be ECS (Embedded Container Service) or OCP (OpenShift Container Platform)> use_stored_user:<(optional) Boolean property which can be true or false. Use this property in conjunction with do-as described below.> do_as:<(optional) if the value of use_stored_user is set to true, this value is used as a fallback when the stored user is not valid. Otherwise, this is directly used as job owner. If the use_stored_user parameter is set to false and no value is supplied in the do_as parameter, then no validation will be performed for the job's username and it will be restored as it is.>Example:
$ cat $BASE_WORK_DIR/cde-upgrade-util.properties cdp_k8s_namespace=cdp-test cdp_endpoint=https://console-cdp-test.apps.cluster.example.com credential_file_path=/home/dex/.cdp/credentials de_admin_user=user de_admin_password=cGFzc3dvcmQ= tls_insecure=true auto_unpause_jobs=true platform_type=ECS use_stored_user=true do_as=cdp-test-user -
Make a note of the details of the CDE service that is being migrated. This
information is required if you are using a CDP database that is external and is
not accessible from the container which is running the cde-upgrade endpoint
stability commands. Identify the cluster endpoint:
- In the Cloudera Data Platform (CDP) console, click the Data Engineering tile. The CDE Home page displays.
- Click Administration in the left navigation menu. The Administration page displays.
- In the Services column on the left, click the Cluster Details icon corresponding to the CDE service whose endpoint you want to migrate.
- Make a note of the CDE cluster ID.
- During the restore operation, both old and the new CDE services use the same resources allocated to existing CDE service. Hence, you must double the resource pool size using the Quota Management option. For example, if root.default.sales is the pool that is used for the old or existing CDE service, you must double the CPU and memory resources of this pool. Also, make sure that you have sufficient hardware when doubling the resource pool size.
