October 12, 2022
This release (1.17-h1) of the Cloudera Data Engineering Service on CDP Public Cloud now supports Apache Iceberg 0.14.
Support for Iceberg 0.14
When you upgrade to CDE 1.17-h1, ensure that you also upgrade to Iceberg 0.14. For more information, see Using Apache Iceberg in Cloudera Data Engineering. The following features are included with Iceberg 0.14:
- MERGE operations allow for bulk updates and DELETES.
- CDE Azure deployments are now able to leverage Iceberg for Lakehouse architecture.
- [Technical Preview] Iceberg table format version 2 (v2) is the latest specification
available and includes the following key updates:
- UPDATE and DELETE operations follow the Iceberg format v2 row-level position delete specification and enforce snapshot isolation.
- DELETES, UPDATES, and MERGE operations use the merge-on-read function by default. Merge-on-read is more efficient than the copy-on-write function because it does not rewrite file data.
Set default values for the variables in CDE job specification
Using [--default-variable] flags you can now replace strings in job values. For more information, see Creating and updating Apache Spark jobs using the CLI.