Backing up and restoring Cloudera Data Engineering jobsPDF version

Restoring jobs and Airflow Operators and Libraries from a Cloudera Data Engineering version 1.20.3 VC backup

In Cloudera Data Engineering version 1.20.3, the backup and restore function was not implemented yet for Airflow Operators and Libraries. To restore a Cloudera Data Engineering version 1.20.3 Virtual Cluster (VC) from a backup, first create the VC manually, after that create and activate the Airflow Operators and Libraries with the same requirements.txt file that the original VC used, and restore the jobs, resources, and so on from the archive.

Make sure that you have created a backup on a local storage from a Cloudera Data Engineering version 1.20.3 VC with active state Airflow Operators and Libraries with the following command:

cde backup create --include-jobs --include-resources --include-credential-secrets --include-credentials --local-path backup.zip

Optionally, you can validate the backup.

cde backup list-archive --local-path backup.zip
  1. Create a VC manually with the same configuration (Spark version, CPU, memory, and so on) as the original VC.
    For more information, see Creating virtual clusters.
  2. Create and activate the Airflow Operators and Libraries manually. Use the same requirements.txt file that the backed up VC used.
    For more information about obtaining the requirements.txt file, see In-place upgrade with Airflow Operators and Libraries.
  3. Restore the jobs, resources, and so on from the archive.
    cde backup restore --local-path backup.zip

    Example:

    $ cde backup restore --local-path backup.zip
     8.2KB/8.2KB 100% [==============================================] backup.zip
All items restored successfully! Details:
Successfully restored items:
  Job:helloworld ( Created )
  Job:sparkhello ( Created )
  Resource:files ( Created )
  Resource:cde-airflow-pyenv-1711096070 ( Created )