Handling ineligible upgrades for Cloudera Data Engineering

In certain scenarios, your Cloudera Data Engineering (CDE) service is not eligible for an in-place upgrade to CDE 1.21. In such a scenario, follow a manual upgrade process where administrators create a new service and import all jobs and resources. Learn how to conduct a workaround for each scenario.

Older CDE versions

If the version of your CDE service is older than six months, it is outside of the upgrade window. Older CDE versions do not support in-place upgrade and you cannot use the process listed in Handling upgrade failures for CDE. This table provides information about the supported upgrade versions and the target version.

Target version Supported Upgrade Versions
1.21

Azure: 1.20.3 onwards

AWS: 1.19.2 onwards

Solution: A manual upgrade option is available. To upgrade, create a new CDE Service and transfer your jobs to the new service by following the instructions in Backing up and restoring CDE jobs.

Older Data Lake versions

If your CDE service uses an old Data Lake version, and therefore it is not compatible with the latest CDE version, your CDE service is not eligible to perform either an in-place upgrade or a manual upgrade. For information about CDE and its compatible Data Lake version, see Compatibility for Cloudera Data Engineering and Runtime components.

Solution: Upgrade your Data Lake to a compatible version associated with the CDE version. For more information about upgrading Data Lake, see Upgrading a Data Lake.

Private networking

If you use services that run on a private network, (for example, Private EKS), your CDE service is not eligible for an in-place upgrade.

Solution: A manual upgrade option is available. To upgrade, you must create a new CDE Service and transfer your jobs to the new service by following the instructions in Backing up and restoring CDE jobs.

Airflow operators and libraries

If you use Airflow operators and libraries, before you can use the in-place upgrade, follow the procedure described in In-place upgrade with Airflow Operators and Libraries.

Solution: See In-place upgrade with Airflow Operators and Libraries.

SSD-based clusters on AWS

If you create new CDE services on CDE version 1.19.4 or higher with an SSD instance enabled, your CDE service is not eligible for an in-place upgrade.

Solution: A manual upgrade option is available. To upgrade, create a new CDE service and transfer your jobs to the new service following the instructions in Backing up CDE jobs on remote storage.

Hotfix version of CDE

There is no upgrade available to CDE hotfix versions.