Iceberg
This topic describes the Iceberg-related known issues in Cloudera Data Warehouse (CDW) Private Cloud.
Technical Service Bulletins
- TSB 2024-746: Concurrent compactions and modify statements can corrupt Iceberg tables
- Apache Hive (Hive) and Apache Impala (Impala) modify statements
(
DELETE
/UPDATE
/MERGE
) on Apache Iceberg (Iceberg) V2 tables can corrupt the tables if there is a concurrent table compaction from Apache Spark. The issue happens when the compaction and modify statement run in parallel, and when the compaction job commits before the modify statement. In this case the position delete files of the modify statement still point to the old files. This means the following in case ofDELETE
statements- Deleting records pointing to old files have no effect
UPDATE
/MERGE
statements- Deleting records pointing to old files have no effect
- The table will also have the newly added data records
- Rewritten records will still be active
This issue does not affect Apache NiFi (NiFi) and Apache Flink (Flink) as these components write equality delete files.
- Knowledge article
-
For the latest update on this issue see the corresponding Knowledge article: TSB 2024-746: Concurrent compactions and modify statements can corrupt Iceberg tables