Migrating Cloudera Data Engineering Services deployed on deprecated Redhat UBI images to Security-Hardened images

An optional restore path is available to migrate Virtual Cluster (VC) backups created with deprecated Redhat UBI images to Security-Hardened images. This capability allows administrators to restore backups after the support for insecure Redhat UBI images was removed in Cloudera Data Engineering 1.26.2.

The force_secure_spark_os parameter was added to the restore service API, allowing insecure images to be force-migrated to Security-Hardened images when set to true. By default, this parameter is set to false.

Steps

  1. Add the following payload when calling the restore service API:
    cdpcurl -X 'POST' \
      'CDP_ENDPOINT_URL/dex/api/v1/cluster/backup/restore' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d {"backup_id": [***BACKUP-ID***], "environment_crn": "[***ENVIRONMENT-CRN***]", "cluster_name": "[***CLUSTER-NAME***]", "force_secure_spark_os": true}
    

    The payload uses the following variables:

    • BACKUP-ID - The ID of the backup that you are restoring from. For example: "backup_id": 4
    • ENVIRONMENT-CRN - The Customer Resource Number (CRN) of the Cloudera environment with which a restored Cloudera Data Engineering Service is associated. Currently, you can restore the Cloudera Data Engineering service only to the same Cloudera environment to which the backed-up service is associated. For example: "environment_crn": "crn:cdp:environments:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:environment:2db7823a-c51a-4a0e-b73a-dc5b2a0418d5"
    • CLUSTER-NAME - The name of the restored Cloudera Data Engineering Service. For example: "cluster_name": "test-insecure3"

    Step result: The affected VCs are restored with Security-Hardened images.

Step troubleshooting

  1. Check the job restore logs at ${[***BASE-URL***]}/dex/api/v1/cluster/${[***CLUSTER-ID***]}/initialization-logs to find out what failed to be restored.

    The command uses the following variables:

    • BASE-URL

      Get the base URL:

      • You can obtain the BASE-URL by copying the console URL up to the first forward slash (/).

        Example for the console URL: https://console.us-west-1.cdp.cloudera.com/dex/home

        An example for the BASE-URL, which changes according to the region: https://console.us-west-1.cdp.cloudera.com

      • Alternatively:
        1. In the Cloudera Management Console, click the Data Engineering tile.
        2. In the left navigation menu, click Administration.
        3. Right click on the Administration page and select Inspect from the list of options.
        4. Click the Console tab.
        5. Type window.location.origin and press Enter.
    • CLUSTER-ID

      To obtain the CLUSTER-ID, log in to the Cloudera Data Engineering UI and navigate to Administration in the left-navigation menu. Select the respective Service, then click on the Service Details icon, and copy the CLUSTER ID.

  2. Resolve the failed Jobs case-by-case.