Cloudera Data Engineering upgrade version compatibility

Checking your upgrade options

In-place upgrades are supported from Cloudera Data Services on premises 1.5.5 SP3 including cumulative hotfixes such as 1.5.5 SP3 CHF3 to 1.5.5 SP4. Services running earlier releases are not eligible for in-place upgrades and must use a side-by-side upgrade with the dex-upgrade-utils Docker image. For all supported upgrade paths, see Cloudera Data Services on premises upgrade documentation.

In-place upgrades

To perform an in-place upgrade, your Cloudera Data Services on premises deployment requires an eligible release. The following table lists all releases eligible for upgrade to the latest release.

Side-by-side upgrade for ineligible releases

If your version does not support in-place upgrades, use the side-by-side upgrade with the dex-upgrade-utils Docker image. With this method, you create a new Cloudera Data Engineering service on the upgraded platform and migrate your jobs, resources, and job run history. For details, see Migrating the Cloudera Data Engineering endpoint and Post-upgrade tasks for the Cloudera Data Engineering endpoint.

Runtime component version behavior after upgrading

To maintain backwards compatibility, the in-place upgrade does not modify the image type. Virtual Clusters that use the older RedHat (deprecated) images continue to use them after the upgrade. To use the new security hardened image after the upgrade, you must create new Virtual Clusters and select the new image. For more information on the supported runtime components, see Compatibility for Cloudera Data Engineering and Runtime components.

Upgrade precheck for version compatibility

As part of the upgrade preparation workflow, a check has been introduced to ensure that the Virtual Clusters (VCs) use a Spark version that is compatible with the Cloudera Base on premises (Cloudera Runtime) version. If the Spark version used for the VCs is incompatible with the Cloudera Runtime version, the upgrade preparation fails. For example, if the Cloudera Runtime version is upgraded from 7.1.9 SP1 to 7.3.1, and any of the VCs use a Spark version lower than 3.5, the upgrade preparation fails.

Historical Job Run logs visibility after in-place upgrade

When performing an in-place upgrade, you may lose visibility of historical job run logs in the Cloudera Data Engineering UI, API. This limitation applies only if the Cloudera Data Engineering service you are upgrading was previously created as a result of an earlier side-by-side upgrade (where jobs, resources, and job run history were migrated into a new service). TThe following logs and UI components are not visible for pre-upgrade job runs:

• Driver logs
• Executor logs
• Submitter logs
• Spark History Server UI

Root cause:

During an earlier side-by-side upgrade, your historical logs were left in a storage location tied to the service that existed at that time. Because an in-place upgrade points the service back to its own native storage location, it no longer references the legacy storage location. Consequently, while your historical logs safely remain in object storage, they are temporarily invisible within Cloudera Data Engineering.

Resolution:

Until a resolution is applied, your historical logs will remain in object storage but stay invisible in Cloudera Data Engineering.

Before initiating the upgrade, perform the following steps:

1. Verify impact: Contact Cloudera Support to confirm whether your specific service is affected by this limitation.
2. Raise a support case: If affected, open a support case ahead of time so your chosen option (Retain or Discard) can be scheduled immediately following the upgrade.
3. Backup critical logs: Download and save any essential historical logs outside of Cloudera Data Engineering that you need to guarantee access to.

After the upgrade completes, contact the Cloudera Support to perform one of the following two options: