Backing up Cloudera Data Engineering jobs

You can back up Cloudera Data Engineering (CDE) jobs and associated resources. Backups are saved as zip files that can be used to restore jobs and their associated resources.

CDE CLI
CDE API
Web UI

Before you begin

Download and configure the CDE CLI.

Steps

Run the cde backup create command to create a backup of the jobs in the virtual cluster your CDE CLI is configured to interact with. By default, all job configurations in the cluster are backed up, but the resources are not. You can use command flags to change this behavior as follows:

--include-job-resources

Backs up all resources associated with the selected jobs. These resources cannot be filtered out by the --resource-filter parameter.

--include-jobs

Backs up all jobs in the virtual cluster. This is the default behavior.

--include-resources

Backs up all resources in the virtual cluster, including those not associated with jobs.

--job-filter <filter_string>

Selects jobs to back up using the fieldname[operator]argument syntax. This command flag can be repeated.

--resource-filter <filter_string>

Selects resources to back up using the fieldname[operator]argument syntax. This command flag can be repeated.

--local-path <filename>

Specifies the local file path and name to store the backup. By default, the backup filename is archive-<timestamp>.zip.

For example, to backup all jobs containing the string etl and include all resources associated with those jobs:
```
cde backup create --job-filter "name[like]%etl%" --include-job-resources
```
Validate the backup using the cde backup list-archive command. For example:
```
cde backup list-archive --local-path archive-2021-11-10T01:24:06.zip
```
Confirm that all jobs and resources that you expected to be backed up are included.

Result

The output of cde backup list-archive is similar to the following:

[
  {
    "entityType": "Resource",
    "cdeVersion": "1.13.0-b199",
    "clusterID": "cluster-n5kmqmwr",
    "appID": "dex-app-47m2j5qz",
    "name": "etl-jars",
    "adjustedName": "etl-jars",
    "archiveDirectoryPath": "v1/resources/v1/709c2910-9695-4496-8f18-5d39a8771c7d/",
    "user": "psherman",
    "backupDate": "2021-11-10T01:24:06.34440405Z"
  },
  {
    "entityType": "Job",
    "cdeVersion": "1.13.0-b199",
    "clusterID": "cluster-n5kmqmwr",
    "appID": "dex-app-47m2j5qz",
    "name": "etl-demo",
    "adjustedName": "etl-demo",
    "archiveDirectoryPath": "v1/jobs/v1/507961fd-f787-4b7c-87d7-7b441bbade73/",
    "user": "psherman",
    "backupDate": "2021-11-10T01:24:06.34440405Z"
  }
]

Before you begin

Request an access token and save it as an environment variable to use in API calls. For instructions, see Getting a Cloudera Data Engineering API access token.

Steps

Determine the API URL for the virtual cluster containing the job you want to back up:
1. Navigate to the Cloudera Data Engineering Overview page.
2. In the CDE Services column, select the service containing the virtual cluster with the jobs you want to back up.
3. In the Virtual Clusters column on the right, click the Cluster Details icon for the virtual cluster containing the jobs you want to back up.
4. Copy the URL under JOBS API URL, and set it as an environment variable. For example:
```
export CDE_JOBS_API="https://pmjkrgn5.cde-czlmkz4y.na-01.xvp2-7p8o.cloudera.site/dex/api/v1"
```

Back up jobs using a URL-encoded filter with the syntax name[like]<query>, modeled after the SQL like operator. For example, to back up jobs containing the string etl, set jobfilter to name[like]%etl% (URL-encoded as name%5Blike%5D%25etl%25):

curl -k \
-H "Authorization: Bearer ${CDE_TOKEN}" \
-X GET "${CDE_JOBS_API}/admin/export?exportjobs=true&jobfilter=name%5Blike%5D%25etl%25&exportjobresources=true&exportresources=false" \
-H "accept: application/zip" \
--output cde-etl-jobs-backup.zip

To back up all jobs and associated resources, omit the jobfilter parameter:

curl -k \
-H "Authorization: Bearer ${CDE_TOKEN}" \
-X GET "${CDE_JOBS_API}/admin/export?exportjobs=true&exportjobresources=true&exportresources=false" \
-H "accept: application/zip" \
--output cde-all-jobs-backup.zip

(Optional) You can validate a backup file by uploading it to the /admin/list-archive endpoint. For example, for a backup file named cde-all-jobs-backup.zip:

curl -k \
-H "Authorization: Bearer ${CDE_TOKEN}" \
-X POST "${CDE_JOBS_API}/admin/list-archive" \
-H "accept: application/json" \
-H "Content-Type: multipart/form-data" \
-F "file=@/path/to/cde-all-jobs-backup.zip;type=application/zip" \
| jq

Before you begin

Steps

Go to the Cloudera Data Engineering Overview page by clicking the Data Engineering tile in the Cloudera Data Platform (CDP) management console.
In the CDE Services column, select the service containing the virtual cluster with the jobs you want to back up.
In the Virtual Clusters column on the right, click the View Jobs icon for the virtual cluster with the jobs you want to back up.
Click Jobs in the left menu.
Click the vertical ellipses menu at the top right, and then click Backup Jobs.

Result

Depending on your browser settings, you are either prompted for a location to save the file, or the file is downloaded to your default download location. The file is a ZIP file named archive-<timestamp>.zip.

To restore a backup file, see Restoring Cloudera Data Engineering jobs from backup.