Backing up Cloudera Data Engineering jobs

You can back up Cloudera Data Engineering (CDE) jobs and associated resources using the API. Backups are saved as zip files that can be used to restore jobs and their associated resources. Currently, backup and restore operations are supported using the API only.

Request an access token and save it as an environment variable to use in API calls. For instructions, see Getting a Cloudera Data Engineering API access token.

  1. Determine the API URL for the virtual cluster containing the job you want to back up:
    1. Navigate to the Cloudera Data Engineering Overview page.
    2. In the Environments column, select the environment containing the virtual cluster you want to interact with using the API.
    3. In the Virtual Clusters column on the right, click the Cluster Details icon for the virtual cluster containing the job to back up.
    4. Copy the URL under JOBS API URL, and set it as an environment variable.
      For example:
      export CDE_JOBS_API="https://pmjkrgn5.cde-czlmkz4y.na-01.xvp2-7p8o.cloudera.site/dex/api/v1"
  2. Back up jobs using a URL-encoded filter with the syntax name[like]<query>, modeled after the SQL like operator.
    For example, to back up jobs containing the string etl, set jobfilter to name[like]%etl% (URL-encoded as name%5Blike%5D%25etl%25):
    curl -k \
    -H "Authorization: Bearer ${CDE_TOKEN}" \
    -X GET "${CDE_JOBS_API}/admin/export?exportjobs=true&jobfilter=name%5Blike%5D%25etl%25&exportjobresources=true&exportresources=false" \
    -H "accept: application/zip" \
    --output cde-etl-jobs-backup.zip

    To back up all jobs and associated resources, omit the jobfilter parameter:

    curl -k \
    -H "Authorization: Bearer ${CDE_TOKEN}" \
    -X GET "${CDE_JOBS_API}/admin/export?exportjobs=true&exportjobresources=true&exportresources=false" \
    -H "accept: application/zip" \
    --output cde-all-jobs-backup.zip
  3. (Optional) You can validate a backup file by uploading it to the /admin/list-archive endpoint.
    For example, for a backup file named cde-all-jobs-backup.zip:
    curl -k \
    -H "Authorization: Bearer ${CDE_TOKEN}" \
    -X POST "${CDE_JOBS_API}/admin/list-archive \
    -H "accept: application/json" \
    -H "Content-Type: multipart/form-data" \
    -F "file=@/path/to/cde-all-jobs-backup.zip;type=application/zip" \
    | jq
To restore a backup file, see Restoring Cloudera Data Engineering jobs from backup.