Cloudera Docs

Restoring a Cloudera AI Workbench

Restoring a backup creates a new Cloudera AI Workbench, and recreates all of the projects, jobs, applications and so on in the original workbench.

Restoring a workbench with multiple CPU and GPU resource groups

During a restore operation, the following is the behavior if you have multiple CPU and GPU resource groups:

  • If the backed-up workbench originally had only a single CPU and/or single GPU resource group, the user is restricted to provisioning only single CPU and GPU resource groups during the restore process. The UI will not show the option for multiple groups.
  • If the backed-up workbench already contained multiple CPU/GPU resource groups, the customer is free to restore to a multi CPU/GPU environment.
  • Rerunning Workloads: If you re-run an existing workload created prior to the upgrade, it will continue to use the resources it was previously assigned. If you want that workload to run on a newly created Resource Group, you must explicitly select the new Resource Group when re-running or re-creating the workload.
  1. In the Workbench Backups UI, find the workbench to restore. You can search for the workbench name or CRN. There can be multiple backups for a given workbench.
  2. Enter the workbench, and manually stop all workloads (sessions, jobs, applications, and models).
    For external NFS backed workbenches, copy the manual backup of external NFS data (corresponding to this particular backup) to the configured external NFS export. Ignore this step if the workbench is configured with internal NFS, as internal NFS data is backed up and restored automatically by Cloudera AI.
  3. Look for the backup to restore, and click Restore. The restore process starts, and the workplace states changes to Creating Workbench.
  4. Provision a new workbench that is in the same Cloudera environment as the original workbench.
The restore process can take some time, depending on the amount of data to copy. When it is complete, you can find the restored workbench in the Workbenches UI.

Monitoring event logs

You can monitor the progress of the backup process by checking the event logs. In the Actions menu for the workbench, click View Event Logs, and then on the Events & Logs tab, click View Event Logs again for the latest backup event.

When the backup process completes, the workbench enters the installation completed state again.

If there were issues during backup, appropriate error messages will be displayed in the event logs. However the workbench will recover from failure and will be reverted back to the original state when backup was triggered.