Cloudera Docs

Prerequisites for upgrading CDE Service with endpoint stability

You must first download the docker image and create the cde-upgrade-util.properties file to back up Cloudera Data Engineering (CDE) services.

Ensure that the host meets the following conditions:
  • Docker must be running in the host.
  • OCP Kubernetes APIs must be reachable from this host
  • The CDP Control Plane must be reachable from this host.
  1. Download the dex-upgrade-utils docker image tarball. The file naming convention is dex-upgrade-utils-[***VERSION-NUMBER***]-[***BUILD-NUMBER***].tar.gz.
  2. Load the downloaded image into the host machine docker runtime: 
    docker load < dex-upgrade-utils-[***VERSION-NUMBER***]-[***BUILD-NUMBER***].tar.gz

    Example:

    docker load < dex-upgrade-utils-1.20.1-b48.tar.gz
  3. Create a directory in the host machine and export that path as BASE_WORK_DIR. 
    mkdir [***HOST_MACHINE_PATH***]
export BASE_WORK_DIR=[***HOST_MACHINE_PATH***]

    Example: 

    mkdir /opt/backup-restore
export BASE_WORK_DIR=/opt/backup-restore
  4. Create backup and secrets directories in the BASE_WORK_DIR directory and update the access permissions. The secrets directory stores the kubeconfig and CDP DE Admin credentials files. The backup directory will store the backup file which will be generated when you backup the CDE Service. 
    cd $BASE_WORK_DIR
mkdir backup secrets
chmod 775 backup
  5. Place the CDP credentials file of the DEAdmin user and administrator kubeconfig file in the $BASE_WORK_DIR/secrets directory.
  6. Create the cde-upgrade-util.properties file as follows:
    1. Create the cde-upgrade-util.properties file and save it in the $BASE_WORK_DIR directory.
    2. Update the following information in the cde-upgrade-util.properties file: 
      cdp_k8s_namespace:<CDP control plane k8s namespace>
cdp_endpoint:<CDP control plane endpoint>
credential_file_path:<Path to the DEAdmin user CDP credentials file>
de_admin_user:<DEAdmin user-id>
de_admin_password:<DEAdmin user's password must be in base64 encoded format. Use the "echo -n [***PASSWORD***] | base64" command to encode the password. >
tls_insecure:<Keep it true if you are using a self-signed certificate>
auto_unpause_jobs: <Specify it as “true” if you want to automatically resume the jobs that were paused during the backup phase. The jobs will be resumed after you restore the CDE service.>
platform_type:OCP
use_stored_user:<(optional) Boolean property which can be true or false. Use this property in conjunction with do-as described below.>
do_as:<(optional) if the value of use_stored_user is set to true, this value is used as a fallback when the stored user is not valid. Otherwise, this is directly used as job owner. If the use_stored_user parameter is set to false and no value is supplied in the do_as parameter, then no validation will be performed for the job's username and it will be restored as it is.>

      For example: The following options are the minimum recommended options that you must include in the cde-upgrade-util.properties file:

      cdp_k8s_namespace=cdp
cdp_endpoint=https://console-cdp.apps.host-1.ecs-pvc1.kcloud.cloudera.com
credential_file_path=/home/dex/.cdp/credentials
de_admin_user=cdpuser1
de_admin_password=VGVzdDEyMw==
tls_insecure=true
auto_unpause_jobs=true
platform_type=OCP
user_stored_user=false
  7. Make a note of the details of the CDE service that is being migrated. This information is required if you are using a CDP database that is external and is not accessible from the container which is running the cde-upgrade endpoint stability commands. Identify the cluster endpoint:
    1. In the Cloudera Data Platform (CDP) console, click the Data Engineering tile. The CDE Home page displays.
    2. Click Administration in the left navigation menu. The Administration page displays.
    3. In the Services column on the left, click the Cluster Details icon corresponding to the CDE service whose endpoint you want to migrate.
    4. Make a note of the CDE cluster ID.