CML service with Data Lake upgrades
CDP environments have two services which can be upgraded individually, the FreeIPA service and the Data Lake (DL) service. CML workspaces run in CDP environments. The FreeIPA service provides identity management, and the Data Lake service provides SDX capabilities to CML workspaces.
In this document we provide FAQs for the behavior of CML workspaces during a Data Lake upgrade. For information on FreeIPA upgrades, see Upgrade FeeIPA.
What kinds of DL upgrades are possible?
The Data Lake service supports the following upgrades.
- Hotfix upgrades
- Version upgrades
- OS version upgrades
These upgrades can all be done from the CDP Data Lake service UI, or with the CDP CLI. DL upgrades require downtime. DL upgrades preserve the state of the data lake during the upgrade.
During DL upgrades, the shape of the data lake cannot change. For example, we cannot change a LIGHT_DUTY shape to a MEDIUM_DUTY_HA shape during the DL upgrade.
What is DL migration and is it supported?
To change the DL shape, perform a DL migration, also called DL scaling. For example, a DL migration can change a DL LIGHT_DUTY shape to a MEDIUM_DUTY_HA shape.
DL migration is currently available as a technical preview. Speak to your Cloudera account team to discuss whether it may be suitable for your situation.
What happens if the DL upgrade/migration fails?
There is no automated backup and restore process for the DL. It is recommended to perform a backup of the DL before starting the upgrade process. If the DL upgrade fails, the recommended option is to delete the failed DL using the CDP CLI (cdp datalake delete-datalake --datalake-name <dl name>) and recreate the DL using the cdpcli. Once the DL is recreated, you need to restore the DL state from the backup. For more information, refer to Backup and Restore for the Data Lake.
Do not delete the environment service during a failed DL upgrade process. Deleting an environment causes all CML workspace running in this environment to be unusable.
Can the Environment service be deleted and recreated at any point if the DL upgrade or migration process has an error?
No. Environments with experiences running inside them cannot be deleted at any time. If you delete the environment, then all the experiences (such as a CML workspace) need to be deleted.
Unless a CML workspace is first backed up and restored, then all state information is lost and you need to start from a fresh workspace. For more information, see Backing up ML workspaces.
What are the CML workspace prerequisites for DL upgrades/migrations?
- Upgrade CML workspaces to the latest version (if an upgrade is available).
- Stop any jobs, sessions, experiments or any workloads that need DL access before performing a DL upgrade.
- Announce to the team that there will be planned downtime for CML workspaces during the DL upgrade process.
Are the CML workspaces operational during DL upgrades/migrations?
It is recommended to NOT use CML workspaces during DL upgrades.
However, the observed behavior of CML workspaces during DL upgrades is as follows.
- CML workspaces remain accessible during DL upgrades. Users can log in to a CML workspace.
- Users can launch sessions, run jobs, experiments, models, and so on which do not require DL access. For example, jobs that do not require IDBroker or SDX/HMS access will function normally.
- Any compute instance that requires IDBroker or SDX/HMS access will fail.
- Any scheduled jobs that require IDBroker or SDX access will fail.
Are any changes to CML workspaces needed after a DL is upgraded or migrated successfully?
No. No further actions are required on a CML workspace after a successful DL upgrade. CML workspaces continue to function normally. Make sure to announce to the team that they can start using CML workspaces as usual.
