Upgrading ML Workspaces

This topic describes how to upgrade existing ML workspaces. Currently, only CDP users with both the MLAdmin role and the EnvironmentAdmin account role can create, upgrade, or remove workspaces.

Existing ML workspaces periodically should be upgraded. Upgrading the workspace upgrades the CML software version to the current version, and may also upgrade cluster software. In case the underlying Kubernetes software must be upgraded, a warning banner displays, notifying you that you should upgrade the workspace promptly.

During an upgrade, any running models and applications shut down, but they automatically restart after the upgrade is complete.
To upgrade Kubernetes, only use the upgrade method provided in CML. Do not upgrade Kubernetes directly in the cloud console or through the CLI. Follow the instructions here to upgrade Kubernetes. If there is some error, then repeat the instructions. This applies to both Microsoft Azure and AWS.
You should back up your workspace before starting the upgrade. For more information, see Backing up ML workspaces.

When is an upgrade necessary?

Cloud service providers define their generally available version of Kubernetes based on their Kubernetes version support policies. For AKS refer to Supported Kubernetes versions in Azure Kubernetes Service (AKS) and for EKS refer to Amazon EKS Kubernetes release calendar.

Cloud service providers may have different deprecation policies for Kubernetes versions:

For AWS deprecation policy, refer their FAQ section in Amazon EKS version support and FAQ.
For Azure, refer to the Azure Kubernetes FAQ.

If any Kubernetes version used in your ML workspaces is deprecated by the cloud providers and CML upgrades are enabled, the warning banner displays.

ACTION REQUIRED: A new CML version is available and it is highly recommended to upgrade
    to the latest version as soon as possible. To perform an upgrade, select Upgrade
     Workspace from the Actions menu.

In order to avoid unplanned service interruption caused by the automatic Kubernetes upgrade by EKS and continue to receive support from AKS for your ML workspaces on Azure, it is important to make sure that your ML workspaces are using supported Kubernetes versions. Upgrading a ML workspace will automatically upgrade the Kubernetes to a supported version. We recommend our users to upgrade the ML workspaces promptly when the warning banner appears.

What type of upgrades does CML Support?

In-place ML upgrades

Upgrades are done in-place on the existing CML workspace. This may involve a Kubernetes upgrade (if there is an upgrade available) followed by upgrading the CML software.

Log in to the CDP web interface.
Click ML Workspaces.
For a given workspace, click on the Actions icon and select Upgrade Workspace.
Click OK to confirm.

The upgrade process may take anywhere between two to four hours, approximately.

Upgrades through CML Backup & Restore

If a CML workspace upgrade from a specific version could not be validated due to Kubernetes version deprecations on cloud providers or is deemed risky, in-place upgrades will be disabled for these versions.

In such cases, depending on the version of CML either the upgrade button is disabled or the in-place upgrade pre-flight check will fail, with a failure message pops up that says:

In-place upgrades from <existing_version> are not supported. Follow the documentation for the backup based upgrade steps.

In this case, it is recommended to go with Backup/Restore to upgrade to the latest CML version, essentially performing a workspace upgrade with all your previous data in place. Refer to ML Upgrades using Backup/Restore for more information.