ML Upgrades using Backup/Restore

Backing up AWS workspace

For information on backing up workspaces, see Backing up ML workspaces.

The time required to backup or restore AWS based workspaces mainly depends on the size of EFS (File System for projects storage) associated with the workspace. To get the EFS ID associated with the workspace, click on View Workspace Details from the UI and note the Filesystem ID. The size associated with the EFS can be retrieved from the AWS console using the Filesystem ID.

The Backup timeout for storage volumes (EFS and EBS) is by default set to 12 hours, but the user can customize the timeout. While clicking on the Backup Workspace, there will be an option to specify Backup Timeout (in minutes), to accommodate users with large EFS size.

Similarly, if the user suspects that the EFS is too large to be restored within the default 12 hours, custom restore timeouts can be set in the advanced settings during restoring an AWS workspace from a backup snapshot through the UI.

Tracking running backups on AWS

For AWS based backup jobs, CML prints out backup job completion percentage for all backup jobs associated with a backup snapshot at an interval of 15 minutes as part of event logs. However, the completion percentage is an estimate returned by AWS APIs, and CML just surfaces the same. CML is not running any mechanisms/heuristics to calculate the completion percentage for a particular backup.

From our experience, we have seen that AWS-provided completion percentages can vary wildly, jump abruptly and can be downright misleading. CML advises to take the percentage numbers with a grain of salt.

Canceling long running backups on AWS

Backups generally take a long time if you have lots of data in EFS. This is expected behavior. However if the backup is taking longer than expected, you can cancel the backup jobs from AWS dashboard and CML will detect this in a while, and will fail the backup.

To cancel the corresponding AWS backup jobs from AWS dashboard:

• Go to AWS Backup > Jobs > Backup Jobs and identify the running backups associated with the backup snapshot. The backups should have started at approximately the same time you triggered the backup from the CML console.
• Abort all such backup jobs.

You can retry the backups again using the above mentioned steps for backing up the workspace.

Backing up Azure workspace

Prerequisites

There are a few prerequisites to Azure Backup/Restore:

• If your environment is configured with a pre-existing resource group, then CML backup service would use the same resource group for taking snapshots of Azure Disks. Else, please ensure that you have a resource group created in your Azure Account with the nomenclature cml-snapshots-<azure_region> . For example, if your Azure workspace resides in the westus2 region, there should be a resource group present named cml-snapshots-westus2.
• Please refer to Azure documentation for roles needed to perform a backup: Use Azure role-based access control to manage Azure Backup recovery points.

Suspending the workspace

Suspend the workspace to ensure correctness of data in NFS on Azure during backup. Suspend the workspace by clicking on the Suspend Workspace option for the workspace and wait for the suspend operation to complete successfully.

Since the workspace is now in a suspended state, it is now guaranteed that no writes/mutations are happening on the NFS or Azure disks associated with the workspace.

Invoking Backup

Once you have the prerequisites sorted, please note the workspace CRN from the View Workspace Details page, and run the following command from CDPCLI to initiate workspace backup. Please replace the values in brackets with your own values.

$ cdp ml backup-workspace --workspace-crn <crn:cdp:ml:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:workspace:792f8cfc-ba33-428d-9c80-b9bc6e799ce9> --backup-name <name-of-backup-for-upgrade>
    
    —-
    {
    "backupCrn": ""crn:cdp:ml:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:workspace_backup:b6cee77e-9e38-4e30-9d72-481088f43de0""
    }
    

Please note the backup CRN returned from the CLI call, as it is the backup snapshot which will be used to restore into a new workspace.

Use these variables to restore into a new workspace:

• backupCRN: The CRN returned in the response of CLI call for backup-workspace
• existingNFS: The existing NFS server path can be retrieved from View Workspace Details > Filesystem ID.
• existingNFSVersion: The existing NFS version can be retrieved from View Workspace Details > NFS Protocol version.

Restore on AWS

For information on this step, see Restore an ML workspace.

The UI for restore is quite similar to the Provision workspace UI and should be familiar.

Restore on Azure

To restore into a new workspace from the backup taken above, please run the following CDP CLI command. The workspace provisioning parameters of the request needs to be configured according to your needs. Please ensure that no “write operations” are undertaken on this restored workspace since we will be using the same NFS in Step 5. This is to ensure that there is no state mismatch between the restored Azure disks and the NFS.

$ cdp ml restore-workspace --cli-input-json ‘{
  "newWorkspaceParameters": {
    "environmentName": "eng-ml-dev-env-azure",
    "workspaceName": "new-workspace",
    "disableTLS": false,
    "usePublicLoadBalancer": false,
    "enableMonitoring": true,
    "enableGovernance": true,
    "enableModelMetrics": true,
    "whitelistAuthorizedIPRanges": false,
    "existingNFS": "<existingNFS>",
    "nfsVersion": "<existingNFSVersion>",
    "provisionK8sRequest": {
      "instanceGroups": [
        {
          "instanceType": "Standard_DS3_v2",
          "rootVolume": {
            "size": 128
          },
          "autoscaling": {
            "minInstances": 1,
            "maxInstances": 10
          }
        }
      ],
      "environmentName": "eng-ml-dev-env-azure",
      "tags": [],
      "network": {
        "topology": {
          "subnets": []
        }
      }
    },
  },
  "backupCrn": "<backupCRN>",
  "useStaticSubdomain": false
}
‘


—-
{
    "workspaceCrn": "crn:cdp:ml:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:workspace:081ee5d2-4e82-487c-9404-2537a0ab4019"
}

Wait for the restore operation to succeed.

After restore, please login to the newly created workspace and verify that all projects from the original workspace are available. Do not launch any sessions or applications, create new projects, or otherwise make any changes to the workspace, as that can make changes to the NFS file system that will be incompatible with what will be restored in step 4 below.

Restore on AWS

To restore a workspace, see Restore as ML workspace. While restoring, please ensure that Use Static Subdomain is checked in the restore UI.

Restore on Azure

To restore into a new workspace from the backup taken above, please run the following CDP CLI command. You need to tune various parameters of the request to suit workspace configuration needs.

$ cdp ml restore-workspace --cli-input-json ‘{
  "newWorkspaceParameters": {
    "environmentName": "eng-ml-dev-env-azure",
    "workspaceName": "new-workspace",
    "disableTLS": false,
    "usePublicLoadBalancer": false,
    "enableMonitoring": true,
    "enableGovernance": true,
    "enableModelMetrics": true,
    "whitelistAuthorizedIPRanges": false,
    "existingNFS": "<existingNFS>",
    "nfsVersion": "<existingNFSVersion>",
    "provisionK8sRequest": {
      "instanceGroups": [
        {
          "instanceType": "Standard_DS3_v2",
          "rootVolume": {
            "size": 128
          },
          "autoscaling": {
            "minInstances": 1,
            "maxInstances": 10
          }
        }
      ],
      "environmentName": "eng-ml-dev-env-azure",
      "tags": [],
      "network": {
        "topology": {
          "subnets": []
        }
      }
    },
  },
  "backupCrn": "<backupCRN>",
  "useStaticSubdomain": true
}
‘


—-
{
    "workspaceCrn": "crn:cdp:ml:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:workspace:081ee5d2-4e82-487c-9404-2537a0ab4019"
}