What's new in Cloudera Data Engineering Private Cloud
This release of Cloudera Data Engineering (CDE) on CDP Private Cloud 1.5.0 includes the following features:
Using spark-submit drop-in migration tool
Cloudera Data Engineering (CDE) now provides a command line tool cde-env to
help migrate your Cloudera Data Platform (CDP) Spark workloads running on CDP Private Cloud
Base (spark-on-YARN) to CDE without having to completely rewrite your existing
For more information, see Using spark-submit drop-in migration tool.
Support for Apache Iceberg (Technical Preview)
- Use tables at petabyte scale without impacting query planning, while benefiting from efficient metadata management, snapshotting, and time-travel.
- CDE supports row level updates via copy-on-write MERGE / UPDATES/ DELETES operations. Copy-on-write is helpful in bulk updates in read heavy use-cases.
For more information, see Using Apache Iceberg.
Updated CDE user interface
The user interface has been updated with easy access to commonly used pages, a new Home page, and a Virtual Cluster drop-down menu that allows you to view relevant content related to each Virtual Cluster that you select. The following user interface changes were made:
- Home: New landing page that displays Virtual Clusters and convenient quick-access links.
- Jobs: Displays jobs for the Virtual Cluster that you select from the drop-down menu in the upper left-hand corner.
- Job Runs: Displays the run history of all jobs within a selected Virtual Cluster.
- Resources: Displays resources created within a selected Virtual Cluster.
- Administration: Displays services and Virtual Clusters that can be customized (previously known as the Overview page.
Support for workload secrets using CLI
CDE now provides a secure way to create and store workload secrets for Cloudera Data Engineering (CDE) Spark Jobs. This is a more secure alternative to storing credentials in plain text embedded in your application or job configuration.
For more information, see Managing workload secrets with CDE Spark Jobs using CDE CLI.
Using multiple profiles
You can now create and use multiple profiles using CDE CLI. You can add a collection of CDE
CLI configurations grouped together as profiles, to the
You can use these profiles while running commands. You can set the configurations either at
a profile level or at a global level.
For more information, see Creating and using multiple profiles using CDE CLI.