November 12, 2024
This release (1.23.0) of the Cloudera Data Engineering service on CDP Public Cloud introduces the following changes.
External IDE connectivity through Spark Connect-based sessions (Technical Preview)
Cloudera Data Engineering supports Spark Connect sessions, which are Cloudera Data Engineering sessions that expose the Spark Connect interface. Spark Connect sessions allow you to run Spark commands from any remote Python environment. For more information, see External IDE connectivity through Spark Connect-based sessions.
In-place upgrade enhancements
In-place upgrade enhancements have been documented to improve handling Airflow version 2.9 and Python version 3.11 changes during the upgrade. For more information, see In-place upgrade with Airflow Operators and Libraries.
Backup-restore-based upgrade enhancements
Backup-restore-based upgrade enhancements have been implemented.
Airflow version upgrade to 2.9
The Airflow version that Cloudera Data Engineering uses is upgraded to 2.9. For more information, see:
Python 3.11 support for Apache Airflow
The Python version that Cloudera Data Engineering uses is upgraded to 3.11. For more information, see Compatibility for Cloudera Data Engineering and Runtime components.
Iceberg version upgrade to 1.5
The Iceberg version that Cloudera Data Engineering uses is upgraded to 1.5.2. For more information, see: Compatibility for Cloudera Data Engineering and Runtime components.
Kubernetes version upgrade to 1.30
The Kubernetes version that Cloudera Data Engineering uses is upgraded to 1.30. For more information, see Compatibility for Cloudera Data Engineering and Runtime components.
Deprecation of CDW Operator
Starting from Cloudera Data Engineering service version 1.23.0, CDW Operator is deprecated. Cloudera recommends you to use SQL Operator in place of CDW Operator. For more information, see Creating a connection to Cloudera Data Warehouse for CDW Operator.
Active node count
The active node count has been removed from the following Cloudera Data Engineering UI pages:
- Administrator Overview
For real-time node count information, check your Cloud Service Provider’s website. For more information, see Checking the node count on your Cloud Service provider’s website.
Fixed issues
- DEX-9617: CDE Jobs Page-Airflow UI: Audit Log tab does not allow further navigation
- The fix allows you to view the Airflow DAG run audit logs.
