Cloudera Docs

Prerequisites and limitations for using Iceberg in Spark

To use Apache Iceberg with Spark, you must be aware of the following limitations:

Limitations

  • Iceberg tables with equality deletes do not support partition evolution or schema evolution on Primary Key columns.

    Users should not do partition evolution on tables with Primary Keys or Identifier Fields available, or do Schema Evolution on Primary Key columns, Partition Columns, or Identifier Fields from Spark.

  • The use of Iceberg tables as Structured Streaming sources or sinks is not supported.
  • PyIceberg is not supported. Using Spark SQL to query Iceberg tables in PySpark is supported.

Iceberg table format version 2

Iceberg table format version 1 and 2 (v1 and v2) is available. Iceberg table format v2 uses row-level UPDATE and DELETE operations that add deleted files to encoded rows that were deleted from existing data files. The DELETE, UPDATE, and MERGE operations function by writing delete files instead of rewriting the affected data files. Additionally, upon reading the data, the encoded deletes are applied to the affected rows that are read. This functionality is called merge-on-read.

With Iceberg table format version 1 (v1), the above-mentioned operations are only supported with copy-on-write where data files are rewritten in their entirety when rows in the files are deleted. Merge-on-read is more efficient for writes, while copy-on-write is more efficient for reads.

Iceberg table format version 3 [Technical Preview]

Starting in Cloudera Runtime 7.3.2 SP1, Spark uses Apache Iceberg library version 1.10.0. Existing Iceberg V1 and V2 table operations are unchanged; you need not run new SQL syntax for current V1 and V2 workflows.

In this release, Spark supports the following Iceberg V3 capabilities only:

  • Deletion vectors — the V3 encoding for position deletes. When you run DELETE on a V3 Iceberg table, Spark records deleted row positions using deletion vectors instead of V2 position delete files. You need not run new SQL syntax to write deletion vectors. Create an Iceberg table with 'format-version'='3' and run a standard DELETE statement.
  • Row lineage tracking — assigns a unique row identifier to each row and records which snapshot last updated a row. This metadata supports change data capture (CDC) use cases. When Spark writes to a V3 table, Spark maintains the row lineage metadata the specification requires; you need not run new SQL syntax to enable row lineage. Row lineage applies to newly written rows on V3 tables. It does not apply to rows updated using equality deletes.

Spark does not support other Iceberg V3 features in this release, including the VARIANT data type, Puffin statistics, geometry and geography types, column default values, nanosecond timestamps, full table encryption, and multi-argument partition transforms. For engine-by-engine V3 support, see the Apache Iceberg Feature Support Matrix.

Replication Manager Iceberg replication policies support Iceberg V1 and V2 tables only. Iceberg V3 tables are not supported for replication in this release.

Example Spark SQL commands to create a V3 table and run row-level deletes:

CREATE TABLE ice_3 (i INT, s STRING) USING iceberg
TBLPROPERTIES ('format-version' = '3');

INSERT INTO ice_3 VALUES (1, 'one'), (2, 'two');
DELETE FROM ice_3 WHERE i = 2;