Cloudera Docs

July 23, 2024

This release (1.22.0) of the Cloudera Data Engineering (CDE) service on CDP Public Cloud introduces the following changes.

AWS Graviton support (GA)

AWS Graviton is a general purpose, ARM-based processor family. From the AWS Graviton family, CDE supports AWS Graviton 3. AWS Graviton delivers currently the best price performance for cloud workloads running in AWS EC2. With AWS Graviton, you can optimize costs and achieve better performance.

For more information, see AWS Graviton instances in CDE.

Support for in-place upgrade with private networking

If you use services that run on a private network, (for example, Private EKS), your CDE service is eligible for an in-place upgrade.

For more information, see In-place upgrade with Apache Airflow Operators and Libraries

Kubernetes version upgrade to 1.28

The Kubernetes version that CDE uses is upgraded to 1.28. For more information, see Compatibility for Cloudera Data Engineering and Runtime components.

Iceberg version upgrade to 1.4.3

The Iceberg version that CDE uses is upgraded to 1.4.3. For more information, see Compatibility for Cloudera Data Engineering and Runtime components.

Support for Spark version 3.5.1

CDE supports Spark version 3.5.1. For more information, see Compatibility for Cloudera Data Engineering and Runtime components.

Virtual Cluster (VC)-level Spark configurations (Technical Preview)

With the VC-level Spark configurations option, you can create and update Spark configurations that, by default, apply to all Spark jobs that are run in a VC.

For more information, see Managing Virtual Cluster-level Spark configurations

Azure private DNS zones (Technical Preview)

With the Azure private DNS zones feature, you can optimize costs by leveraging existing Azure private DNS zone resources. When creating the CDE service, you can use an Azure private DNS zone for the Azure Kubernetes Service (AKS), the Storage Account File Share, and for the database.

For more information, see Azure private DNS zones in a CDE service.

Fixed issues

  • DEX-12763 - Upgraded CDE Service to 1.20.3 and above with SSD Instances unable to run jobs
  • DEX-13755 - Azure and AWS service B&R does not restore all the jobs
  • DEX-13151 - UI keeps loading when jobs are unpaused if the user has readonly access
  • DEX-12634 - VC Security/Access Control allows for duplicate user record
  • DEX-14080 - Remove VC cluster maintenance mode configuration on restore
  • DEX-13154 - View log for VC will not redirect to Logs tab
  • DEX-13742 - UI should send allp details for previous Service version
  • DEX-13262 - UI: editing a schedule for the example job fails to send a request
  • DEX-13385 - UI does not indicate user does not have access to CLI tool
  • DEX-13639 - Without All Purpose details we are showing All Purpose OnDemand and Spot instances on Service Config page
  • DEX-13692 - Include sub seconds into execution date for manually triggered Airflow runs
  • DEX-13958 - In place upgrade fails from 1.19/1.20 -> 1.21/1.22 if custom runtime images are used with public docker registry