Upgrading Cloudera Data EngineeringPDF version

Upgrading Cloudera Data Engineering

Cloudera Data Engineering supports in-place upgrades on both AWS and Azure. Cloudera Data Engineering provides an in-place upgrade option from the eligible versions to the latest version. If you are on a Cloudera Data Engineering version that is not eligible for the in-place upgrade, the in-place upgrade option is disabled and you can perform a backup and restore. The upgrades can be triggered by an Admin from the Cloudera Data Engineering user interface.

To check whether your Cloudera Data Engineering version is eligible for the in-place upgrade option, see Support lifecycle policy.

When you upgrade to the latest version in Cloudera Data Engineering, the upgrade process prepares your service for the upgrade with a click of a button. Your information is also backed up in this process. Learn how to complete the upgrade process here.

The following are not included in the backup:
  • Python-venv resources
  • Airflow custom operators and libraries
  • Airflow connections
  • Airflow variables
  • Logs
    • Job Run logs (Driver, Executor, API)
    • Service logs
    • Virtual Cluster event logs
    • Spark Session logs (Including statement history)
    • Airflow DAG logs
  • Job Runs
    • Job Run history
  • Virtual Cluster end-points
  • Resource: Docker runtimes
Ensure that the catchup option is not enabled for any user's Airflow jobs. Before the backup starts, if the Airflow DAG catchup options are enabled, disable them manually.
  1. In the Cloudera console, click the Data Engineering tile.
    The Cloudera Data Engineering Home page displays.
  2. In the left navigation menu, click Administration, select a Service, and click Upgrade.
  3. Click the Maintenance tab and click Upgrade Service.
  4. Click Start Preparation.
    Cloudera Data Engineering prepares the service for you. If the service cannot be prepared, click Retry or Cancel Upgrade. Canceling the upgrade returns your service to the original state. Optionally, you can click Download Logs to view the Diagnostics page where you can generate a diagnostic bundle. If the preparation is successful, proceed to the next step. If the preparation and cancellation fails, you must contact Cloudera support.
  5. Click Start Backup.

    Cloudera Data Engineering automatically creates a backup for you. The backup is used for restoring if an irrecoverable failure happens during the upgrade.

  6. Optional: When the backup is complete, to view the backup that was created, select Backup Library.
  7. Click Start Upgrade.
  8. After the upgrade is complete, to enable all paused jobs and sessions, click Resume Service.
    The service restarts.
After the upgrade is complete and you resumed the service, the service operates on the upgraded version of Cloudera Data Engineering. If you cancel the upgrade, your service resumes the previous state, which enables all paused jobs and resumes all sessions.

We want your opinion

How can we improve this page?

What kind of feedback do you have?