Apache Iceberg in CDW has some limitations you need to understand.
Iceberg in CDW has the following limitations:
- In CDW version 1.4.2-b118, only V1 tables are supported, although it is possible to create V2 tables. If you use Impala, do not experiment with V2 tables, as described below.
- Legacy Iceberg external tables, created before the CDW version 1.4.1-b86 (released June 22, 2022) from Impala, might be incompatible with other SQL engines.
- You might see differences in Iceberg tables created from Hive and Impala when you run DESCRIBE. In HiveCatalog, an external table created from Impala might not have exactly the same properties as an Iceberg table created from Hive.
- Impala does not support ALTER REPLACE COLUMNS.
- Migrating an Impala table to Iceberg is not supported in this release.
- ADD/DROP PARTITION are not supported since partitions are managed by Iceberg.
- If partition columns are not present in the data files, tables cannot be read.
- If the DATALAKE type is CDW, you cannot access Iceberg tables you create in CDW from Cloudera Data Engineering.
- Some data types, such as AVRO, are not supported for storing Iceberg tables. For more information, see Unsupported data types.
- AWS storage is the only storage supported.
No Iceberg V2 table support
As previously mentioned, only V1 tables are fully supported in CDW. You can actually create V2 tables from Hive and Spark if you set some configuration options. For example, you can specify the table property format-version to create a V2 table from Hive; however, this can cause problems for Impala users.
V2 tables support row-level modifications, but if you delete or update rows, the tables will be unreadable from Impala.