Backups for Cloudera AI Workbenches

Cloudera AI enables the efficient creation of machine learning projects, jobs, experiments, machine learning models, and applications within workbenches. The data and metadata of these artifacts are stored in different types of storage systems in on premises environments or in external NFS-backed workbenches outside of an on premises environment.

You can back up a Cloudera AI Workbench, and restore it at a later time. The backup preserves all files, models, applications and other assets within the workbench. However, for external NFS-based workbenches, files are not automatically backed up by Cloudera AI. All workbench backups are accessible through the Workbench Backup Catalog UI.

The Backup and Restore feature allows you to keep your machine learning artifacts safe by backing up all data, except files in external NFS-backed workbenches, to protect against potential disasters. If your Cloudera AI Workbench is backed up, this feature enables you to restore the saved data allowing you to recover your Cloudera AI artifacts exactly as they were at the time of the backup. Administrators can use the Backup and Restore feature to perform on-demand backups of Cloudera AI Workbenches. During the backup process, core services running in the workbench are temporarily shut down to ensure the consistency of the backup data. To minimize disruptions, Cloudera recommends to schedule backups during off-peak hours.

The duration of the workbench backup process depends on the volume of data being copied. The backup procedure involves transferring data from both block volumes and internal NFS, with the time required to back up NFS typically being the most significant factor. Cloudera recommends to regularly back up Cloudera AI Workbenches to ensure data presercvation.

The time required to backup NFS largely depends on the volume of data, as well as the type and number of files. Based on the data size, you can configure a timeout value during the backup process. The status of ongoing and previous backups can be monitored trough the Cloudera AI Workbench UI and the Backup Catalog UI.

There is no restriction on the number of backups that can be created, and the backup snapshots are retained indefinitely within the underlying on premises cluster as long as the original workbench (from which this backup was taken from) is not deleted. Cloudera AI Workbench backup details are stored in the Workbench Backup Catalog UI within the Cloudera AI Control Plane. These entries can be listed, viewed, deleted or restored as needed.

Restoring a backup overwrites the existing Cloudera AI Workbench, from which this backup was taken, with the data from the backup. During this process, all the projects, jobs, applications, and other assets, that existed during the backup are automatically available in the new workbench. The restoration process involves overwriting the current workbench, followed by launching restore jobs to recreate storage volumes from the backup snapshots. The restore process takes longer than a standard workbench provisioning operation due to the additional steps involved in copying data from the backup to the new storage volumes. Restores are always performed as full-copy operations, which include the restoration of metadata files (such as the Sense database) and project files, as well as a complete restoration of data at the storage level. The restoration time is primarily influenced by the NFS restoration process, which typically takes at least as long as the original backup of the file system. Additionally, the restored workbench is always created using the latest version of the Cloudera AI software, which may differ from the version of the original workbench that was backed up.

Restoring the workbench creates a workbench based on the control plane version. The data will be restored to the state when the backup was created but the workbench version will be the one supported by control plane. Consequently, if the control plane is version 1.5.5, reverting to a previous version is not possible by design. Backups are exclusively intended for disaster recovery, enabling data restoration but not version rollback.