Cloudera AI upgrades using Backup/Restore

Backing up AWS workbench

For information on backing up workbenches, see Backing up Cloudera AI Workbenches.

The time required to backup or restore AWS based workbenches mainly depends on the size of EFS (File System for projects storage) associated with the workbench. To get the EFS ID associated with the workbench, click on View Workbench Details from the UI and note the Filesystem ID. The size associated with the EFS can be retrieved from the AWS console using the Filesystem ID.

The Backup timeout for storage volumes (EFS and EBS) is by default set to 12 hours, but the user can customize the timeout. While clicking on the Backup Workbench , there will be an option to specify Backup Timeout (in minutes), to accommodate users with large EFS size.

Similarly, if the user suspects that the EFS is too large to be restored within the default 12 hours, custom restore timeouts can be set in the advanced settings during restoring an AWS workbench from a backup snapshot through the UI.

Tracking running backups on AWS

For AWS based backup jobs, Cloudera AI prints out backup job completion percentage for all backup jobs associated with a backup snapshot at an interval of 15 minutes as part of event logs. However, the completion percentage is an estimate returned by AWS APIs, and Cloudera AI just surfaces the same. Cloudera AI is not running any mechanisms/heuristics to calculate the completion percentage for a particular backup.

From our experience, we have seen that AWS-provided completion percentages can vary wildly, jump abruptly and can be downright misleading. Cloudera AI advises to take the percentage numbers with a grain of salt.

Canceling long running backups on AWS

Backups generally take a long time if you have lots of data in EFS. This is expected behavior. However if the backup is taking longer than expected, you can cancel the backup jobs from AWS dashboard and Cloudera AI will detect this in a while, and will fail the backup.

To cancel the corresponding AWS backup jobs from AWS dashboard:

• Go to AWS Backup > Jobs > Backup Jobs and identify the running backups associated with the backup snapshot. The backups should have started at approximately the same time you triggered the backup from the Cloudera AI console.
• Abort all such backup jobs.

You can retry the backups again using the above mentioned steps for backing up the workbench.

Backing up Azure workbench

Prerequisites

There are a few prerequisites to Azure Backup/Restore:

• If your environment is configured with a pre-existing resource group, then Cloudera AI backup service would use the same resource group for taking snapshots of Azure Disks. Else, please ensure that you have a resource group created in your Azure Account with the nomenclature cml-snapshots-<azure_region> . For example, if your Azure workbench resides in the westus2 region, there should be a resource group present named cml-snapshots-westus2.
• Please refer to Azure documentation for roles needed to perform a backup: Use Azure role-based access control to manage Azure Backup recovery points.

Suspending the workbench

Suspend the workbench to ensure correctness of data in NFS on Azure during backup. Suspend the workbench by clicking on the Suspend Workbench option for the workbench and wait for the suspend operation to complete successfully.

Since the workbench is now in a suspended state, it is now guaranteed that no writes/mutations are happening on the NFS or Azure disks associated with the workbench.

Invoking Backup

Once you have the prerequisites sorted, please note the workbench CRN from the View Workbench Details page, and run the following command from Cloudera Data PlatformCLI to initiate workbench backup. Please replace the values in brackets with your own values.

$ cdp ml backup-workspace --workspace-crn <crn:cdp:ml:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:workspace:792f8cfc-ba33-428d-9c80-b9bc6e799ce9> --backup-name <name-of-backup-for-upgrade>
    
    —-
    {
    "backupCrn": ""crn:cdp:ml:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:workspace_backup:b6cee77e-9e38-4e30-9d72-481088f43de0""
    }
    

Please note the backup CRN returned from the CLI call, as it is the backup snapshot which will be used to restore into a new workbench.

Use these variables to restore into a new workbench:

• backupCRN: The CRN returned in the response of CLI call for backup-workbench
• existingNFS: The existing NFS server path can be retrieved from View Workbench Details > Filesystem ID.
• existingNFSVersion: The existing NFS version can be retrieved from View Workbench Details > NFS Protocol version.

Restore on AWS

For information on this step, see Restore an Cloudera AI Workbench.

The UI for restore is quite similar to the Provision Workbench UI and shall be familiar.

Restore on Azure

To restore into a new workbench from the backup taken above, please run the following Cloudera Data Platform CLI command. The workbench provisioning parameters of the request needs to be configured according to your needs. Please ensure that no “write operations” are undertaken on this restored workbench since we will be using the same NFS in Step 5. This is to ensure that there is no state mismatch between the restored Azure disks and the NFS.

$ cdp ml restore-workspace --cli-input-json ‘{
  "newWorkspaceParameters": {
    "environmentName": "eng-ml-dev-env-azure",
    "workspaceName": "new-workspace",
    "disableTLS": false,
    "usePublicLoadBalancer": false,
    "enableMonitoring": true,
    "enableGovernance": true,
    "enableModelMetrics": true,
    "whitelistAuthorizedIPRanges": false,
    "existingNFS": "<existingNFS>",
    "nfsVersion": "<existingNFSVersion>",
    "provisionK8sRequest": {
      "instanceGroups": [
        {
          "instanceType": "Standard_DS3_v2",
          "rootVolume": {
            "size": 128
          },
          "autoscaling": {
            "minInstances": 1,
            "maxInstances": 10
          }
        }
      ],
      "environmentName": "eng-ml-dev-env-azure",
      "tags": [],
      "network": {
        "topology": {
          "subnets": []
        }
      }
    },
  },
  "backupCrn": "<backupCRN>",
  "useStaticSubdomain": false
}
‘


—-
{
    "workspaceCrn": "crn:cdp:ml:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:workspace:081ee5d2-4e82-487c-9404-2537a0ab4019"
}

Wait for the restore operation to succeed.

After restore, please login to the newly created workbench and verify that all projects from the original workbenches are available. Do not launch any sessions or applications, create new projects, or otherwise make any changes to the workbench, as that can make changes to the NFS file system that will be incompatible with what will be restored in step 4 below.

Restore on AWS

To restore a workbench, see Restore as Cloudera AI Workbench. While restoring, please ensure that Use Static Subdomain is checked in the restore UI.

Restore on Azure

To restore into a new workbench from the backup taken above, please run the following Cloudera Data Platform CLI command. You need to tune various parameters of the request to suit workbench configuration needs.

$ cdp ml restore-workspace --cli-input-json ‘{
  "newWorkspaceParameters": {
    "environmentName": "eng-ml-dev-env-azure",
    "workspaceName": "new-workspace",
    "disableTLS": false,
    "usePublicLoadBalancer": false,
    "enableMonitoring": true,
    "enableGovernance": true,
    "enableModelMetrics": true,
    "whitelistAuthorizedIPRanges": false,
    "existingNFS": "<existingNFS>",
    "nfsVersion": "<existingNFSVersion>",
    "provisionK8sRequest": {
      "instanceGroups": [
        {
          "instanceType": "Standard_DS3_v2",
          "rootVolume": {
            "size": 128
          },
          "autoscaling": {
            "minInstances": 1,
            "maxInstances": 10
          }
        }
      ],
      "environmentName": "eng-ml-dev-env-azure",
      "tags": [],
      "network": {
        "topology": {
          "subnets": []
        }
      }
    },
  },
  "backupCrn": "<backupCRN>",
  "useStaticSubdomain": true
}
‘


—-
{
    "workspaceCrn": "crn:cdp:ml:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:workspace:081ee5d2-4e82-487c-9404-2537a0ab4019"
}