April 27, 2022

This release (1.15) of the Cloudera Data Engineering (CDE) service on CDP Public Cloud introduces the new features and improvements that are described in this topic.

[Technical Preview] Support for Iceberg 0.13

  • You can use Cloudera Data Engineering virtual clusters running Spark 3 to interact with the latest version (0.13) of Apache Iceberg tables.
  • CDE now supports row level updates via copy-on-write MERGE / UPDATES/ DELETES operations. Copy-on-write is helpful in bulk updates in read heavy use-cases.
  • For more information, see Using Apache Iceberg in Cloudera Data Engineering .

[Technical Preview] In-place upgrades

  • CDE supports upgrades from CDE 1.14 on both AWS and Azure. The upgrades can be triggered by an Admin from CDE UI.
  • Users will need to manually pause/backup/restore each Virtual Cluster to account for upgrade failures.
  • Upgrades of CDE core components include: EKS, AKS Services, and Application Services
  • Upgrades of dependencies include: Helm, K8s versions, YuniKorn

Job email alerts

SLA miss and job failure conditions can be configured for email notifications.

Job runtime notices

Active jobs will now provide notification to the user when certain conditions are met and jobs are not behaving as expected making it easier to understand why jobs might be stuck or not making progress.

For more information, see Running jobs in Cloudera Data Engineering

Spark 3.2

CDE now supports Apache Spark 3.2.

Data Lake upgrades

CDE has now been certified when Data Lake is resized..