Apache Iceberg features

Support for Apache Iceberg library version 1.10.0

Starting with Cloudera Runtime 7.3.2 SP1, Apache Iceberg library version 1.10.0 is supported for Hive and Impala. Existing Iceberg V1 and V2 table operations are unchanged; you do not need new SQL syntax for current workflows.

Variant data type

Hive supports the Iceberg variant data type on Iceberg V3 tables. In Hive, use PARSE_JSON to convert a JSON string into a variant value and VARIANT_GET to extract a nested field using a JSON path and return type.

For more information, see Iceberg data types.

Deletion vector support for Hive

Hive supports deletion vectors on Iceberg V3 tables. Deletion vectors are the Iceberg V3 encoding for position deletes. When you delete rows from a V3 table, Hive records which rows are deleted by position in the referenced data file.

Deletion vectors are a a bitmap of deleted row positions within a single data file. On V3 tables, Hive writes deletion vectors instead of V2 position delete files for new delete operations. Read operations merge active rows with the deletion vectors to return query results. You need not run a new SQL syntax to write deletion vectors. Create an Iceberg table with 'format-version'='3' and run a standard DELETE statement. Hive encodes the delete operation using deletion vectors automatically.

For more information, see Delete data feature and Row-level operations.

Deletion vector support for Impala [Technical Preview]

Impala supports deletion vectors on Iceberg V3 tables. Deletion vectors are the Iceberg V3 encoding for position deletes. When you delete rows from a V3 table from Impala, Impala records which rows are deleted by position in the referenced data file.

On V3 tables, Impala writes deletion vectors instead of V2 position delete files for new delete operations. Read operations merge active rows with the deletion vectors to return query results. You need not run a new SQL syntax to write deletion vectors. Create an Iceberg table with 'format-version'='3' and run a standard DELETE statement. Impala encodes the delete operation using deletion vectors automatically.

For more information, see Delete data feature and Row-level operations.

Row lineage support for Impala [Technical Preview]

Impala supports row lineage tracking on Iceberg format version 3 (V3) tables. Row lineage is an Iceberg V3 capability that assigns a unique row identifier to each row and records which snapshot last updated a row. This metadata supports change data capture (CDC) use cases. To use row lineage from Impala, create an Iceberg V3 table by setting 'format-version'='3'. When Impala writes to the table, Impala maintains the row lineage metadata required; you need not run a new SQL syntax to enable row lineage. Row lineage applies to newly written rows on V3 tables. It does not apply to rows updated using equality deletes.

For more information, see Row lineage feature.

Puffin statistics support for Impala

Impala supports writing Puffin statistics files on Iceberg V3 tables. Puffin is a lightweight container format that stores binary metadata such as deletion vector sidecar files required by the Iceberg V3 specification. To use Puffin statistics from Impala, create an Iceberg V3 table by setting 'format-version'='3'. When Impala performs supported V3 write operations, Impala writes Puffin files as required by the Iceberg V3 specification; you need not run a new SQL syntax to enable Puffin statistics writes.

For more information, see Puffin statistics feature.

Table repair support for Impala

Impala supports the repair_metadata() function for Iceberg tables. Use this function to recover tables that are inaccessible due to missing data files after manual file deletions in the underlying storage. The function synchronizes Iceberg metadata by removing references to missing data files; it does not restore deleted content.

For more information, see Table repair feature.

LIKE predicate pushdown support for Impala

Impala pushes eligible SQL LIKE predicates on Iceberg tables to Iceberg startsWith() and equal() expressions during query planning. This optimization uses Iceberg manifest statistics for file and partition pruning.

For more information, see LIKE predicate pushdown feature.

Supported ACID transaction properties

Iceberg supports atomic and isolated database transaction properties. Writers work in isolation, not affecting the live table, and perform a metadata swap only when the write is complete, making the changes in one atomic commit.

Iceberg uses snapshots to guarantee isolated reads and writes. You see a consistent version of table data without locking the table. Readers always see a consistent version of the data without the need to lock the table. Writers work in isolation, not affecting the live table, and perform a metadata swap only when the write is complete, making the changes in one atomic commit.

Iceberg partitioning

The Iceberg partitioning technique has performance advantages over conventional partitioning, such as Apache Hive partitioning. Iceberg hidden partitioning is easier to use. Iceberg supports in-place partition evolution; to change a partition, you do not rewrite the entire table to add a new partition column, and queries do not need to be rewritten for the updated table. Iceberg continuously gathers data statistics, which supports additional optimizations, such as partition pruning.

Iceberg uses multiple layers of metadata files to find and prune data. Hive and Impala keep track of data at the folder level and not at the file level, performing file list operations when working with data in a table. Performance problems occur during the execution of multiple list operations. Iceberg keeps track of a complete list of files within a table using a persistent tree structure. Changes to an Iceberg table use an atomic object/file level commit to update the path to a new snapshot file. The snapshot points to the individual data files through manifest files.

The manifest files track several data files across many partitions. These files store partition information and column metrics for each data file. A manifest list is an additional index for pruning entire manifests. File pruning increases efficiency.

Iceberg relieves Hive metastore (HMS) pressure by storing partition information in metadata files on the file system/object store instead of within the HMS. This architecture supports rapid scaling without performance hits.