November 23, 2022
This release (1.18) of the Cloudera Data Engineering Service on CDP Public Cloud introduces new features and improvements that are described in this topic.
Updated CDE user interface
- Left-hand menu displays the following:
- Home- New landing page that displays Virtual Clusters and convenient quick-access links.
- Jobs - Displays jobs for the Virtual Cluster that you select from the drop-down menu in the upper left-hand corner.
- Job Runs - Displays the run history of all jobs within a selected Virtual Cluster.
- Resources - Displays resources created within a selected Virtual Cluster.
- Administration - Displays services and Virtual Clusters that can be customized (previously known as the Overview page.
Airflow scaling improvements include support for 1500 DAGs on AWS and about 300 to 500 DAGs when deploying on Azure. For more information, see Apache Airflow scaling and tuning considerations.
Support for the eu-1 (Germany) and ap-1 (Australia) regional Control Plane
The eu-1 (Germany) and ap-1 (Australia) regional Control Plane now supports CDE. For the list of all supported services for all supported Control Plane regions, see CDP Control Plane regions.
Support for workload secrets using API
CDE now provides a secure way to create and store workload secrets for Cloudera Data Engineering (CDE) Spark Jobs. This is a more secure alternative to storing credentials in plain text embedded in your application or job configuration. For more information, see Managing workload secrets with Cloudera Data Engineering Spark Jobs using API.
Java Virtual Machine Debugger (Tech preview)
Attaching a remote debugger (Java virtual machine (JVM) debugger) to a CDE Spark job is now supported as a technical preview feature. For more information, see Using Java virtual machine (JVM) debugger with Apache Spark jobs in Cloudera Data Engineering (Preview) .
Hive Warehouse Connector tables
Hive Warehouse Connector (HWC) tables are now supported in Spark 3 of CDE.
Backup & Restore in object storage
Remote backup storage (object store) is now supported. Previously, only backup to and restore from local storage was supported. This is supported through the CLI and API only. For more information, see Backing up Cloudera Data Engineering jobs and Restoring Cloudera Data Engineering jobs from backup.
Limitations for raw Scala code in CDE
Limitations have been added to the raw Scala code. For limitation details, see Running raw Scala code in Cloudera Data Engineering.
Support for Iceberg V2
- UPDATE and DELETE operations follow the Iceberg format v2 row-level position delete specification and enforces snapshot isolation.
- DELETES, UPDATES, and MERGE operations use the merge-on-read function by default. Merge-on-read is more efficiant than the copy-on-write function because it does not rewrite file data.
For more information, see Prerequisites