Cloudera Docs

What's new in Cloudera Data Engineering Private Cloud

This release of Cloudera Data Engineering (CDE) on CDP Private Cloud 1.5.0 includes the following features:

Using spark-submit drop-in migration tool

Cloudera Data Engineering (CDE) now provides a command line tool cde-env to help migrate your Cloudera Data Platform (CDP) Spark workloads running on CDP Private Cloud Base (spark-on-YARN) to CDE without having to completely rewrite your existing spark-submit command-lines.

For more information, see Using spark-submit drop-in migration tool.

Support for Apache Iceberg (Technical Preview)

Apache Iceberg tables are now supported with Spark 3 virtual clusters.
  • Use tables at petabyte scale without impacting query planning, while benefiting from efficient metadata management, snapshotting, and time-travel.
  • CDE supports row level updates via copy-on-write MERGE / UPDATES/ DELETES operations. Copy-on-write is helpful in bulk updates in read heavy use-cases.

For more information, see Using Apache Iceberg.

Updated CDE user interface

The user interface has been updated with easy access to commonly used pages, a new Home page, and a Virtual Cluster drop-down menu that allows you to view relevant content related to each Virtual Cluster that you select. The following user interface changes were made:

Left-hand menu displays the following:
  • Home: New landing page that displays Virtual Clusters and convenient quick-access links.
  • Jobs: Displays jobs for the Virtual Cluster that you select from the drop-down menu in the upper left-hand corner.
  • Job Runs: Displays the run history of all jobs within a selected Virtual Cluster.
  • Resources: Displays resources created within a selected Virtual Cluster.
  • Administration: Displays services and Virtual Clusters that can be customized (previously known as the Overview page.

Support for workload secrets using CLI

CDE now provides a secure way to create and store workload secrets for Cloudera Data Engineering (CDE) Spark Jobs. This is a more secure alternative to storing credentials in plain text embedded in your application or job configuration.

For more information, see Managing workload secrets with CDE Spark Jobs using CDE CLI.

Using multiple profiles

You can now create and use multiple profiles using CDE CLI. You can add a collection of CDE CLI configurations grouped together as profiles, to the config.yaml file. You can use these profiles while running commands. You can set the configurations either at a profile level or at a global level.

For more information, see Creating and using multiple profiles using CDE CLI.