April 27, 2022

This release (1.15) of the Cloudera Data Engineering (CDE) service on CDP Public Cloud introduces the following changes.

[Technical Preview] Support for Iceberg 0.13🔗

You can use Cloudera Data Engineering virtual clusters running Spark 3 to interact with the latest version (0.13) of Apache Iceberg tables.
CDE now supports row level updates via copy-on-write MERGE / UPDATES/ DELETES operations. Copy-on-write is helpful in bulk updates in read heavy use-cases.
For more information, see Using Apache Iceberg in Cloudera Data Engineering .

[Technical Preview] In-place upgrades 🔗

CDE supports upgrades from two CDE versions prior for both AWS and Azure. For example, if the current CDE version is 1.18, then upgrades are supported from CDE 1.16. The upgrades can be triggered by an Admin from CDE UI.
Users will need to manually pause/backup/restore each Virtual Cluster to account for upgrade failures.
Upgrades of CDE core components include: EKS, AKS Services, and Application Services
Upgrades of dependencies include: Helm, K8s versions, YuniKorn
For more information, see CDE In-place Upgrades (Preview)

Job email alerts🔗

SLA miss and job failure conditions can be configured for email notifications.

Job runtime notices🔗

Active jobs will now provide notification to the user when certain conditions are met and jobs are not behaving as expected making it easier to understand why jobs might be stuck or not making progress.

For more information, see Running jobs in Cloudera Data Engineering

Spark 3.2🔗

CDE now supports Apache Spark 3.2.

Data Lake upgrades🔗

CDE has now been certified when Data Lake is resized..