What's New in Apache Iceberg

Cloudera Runtime 7.3.2.10000 SP1

Upgrade Hive Iceberg library to Apache Iceberg 1.10.0

In Cloudera Runtime 7.3.2 SP1, the Apache Iceberg library used by the Hive engine is upgraded to Apache Iceberg 1.10.0. Existing Iceberg V1 and V2 table operations are unchanged; you need not run a new SQL syntax for current workflows.

This upgrade aligns Hive with the platform-wide Iceberg 1.10.0 library and provides the foundation for Iceberg V3 capabilities in Hive. For supported Iceberg features, see the Apache Iceberg Feature Support Matrix.

Apache Jira: HIVE-28665

Upgrade Impala Iceberg library to Apache Iceberg 1.10.0

In Cloudera Runtime 7.3.2 SP1, the Apache Iceberg library used by the Impala engine is upgraded to Apache Iceberg 1.10.0. Existing Iceberg V1 and V2 table operations are unchanged; you need not run a new SQL syntax for current workflows.

This upgrade aligns Impala with the platform-wide Iceberg 1.10.0 library and provides the foundation for Iceberg V3 capabilities in Impala. For supported Iceberg features, see the Apache Iceberg Feature Support Matrix.

Apache Jira: IMPALA-14665

Hive Iceberg variant data type support

Hive supports the Iceberg VARIANT data type. The Apache Iceberg format specification defines this semi-structured type for Iceberg format version 3. You can store semi-structured data in VARIANT columns and query nested fields using PARSE_JSON and VARIANT_GET (HIVE-29183).

For more information, see Iceberg data types and Create table feature.

Apache Jira: HIVE-29183

Hive Iceberg deletion vector support

Hive supports deletion vectors on Iceberg V3 tables. Deletion vectors are the Iceberg V3 encoding for position deletes and replaces the V2 position delete files for new delete operations on V3 tables. When you run DELETE on a V3 Iceberg table, Hive records deleted row positions using deletion vectors.

For more information, see Delete data feature and Row-level operations.

Apache Jira: HIVE-29006

Impala Iceberg deletion vector support [Technical Preview]

Impala supports deletion vectors on Iceberg V3 tables. Deletion vectors are the Iceberg V3 encoding for position deletes and replace V2 position delete files for new delete operations on V3 tables. Impala can read tables that contain deletion vectors written by Impala or by other Iceberg V3-compatible engines. When you run DELETE on a V3 Iceberg table from Impala, Impala records deleted row positions using deletion vectors.

For more information, see Delete data feature and Row-level operations.

Apache Jira: IMPALA-14585

Impala Iceberg row lineage support [Technical Preview]

Impala supports row lineage tracking on Iceberg format V3 tables. Row lineage is an Iceberg V3 capability that assigns a unique identifier to each row and records row provenance across table snapshots. This metadata supports change data capture (CDC) use cases. To use row lineage from Impala, create an Iceberg V3 table by setting 'format-version'='3'. When Impala writes to the table, Impala maintains the row lineage metadata required; you need not run a new SQL syntax to enable row lineage.

For more information, see Row lineage feature.

Apache Jira: IMPALA-14592

Impala Iceberg puffin statistics support

Impala supports writing Puffin statistics files for Iceberg V3 tables. Puffin is a lightweight container format that stores binary metadata such as deletion vector sidecar files required by the Iceberg V3 specification.

For more information, see Puffin statistics feature.

Apache Jira: IMPALA-14755

Impala Iceberg table repair support

Impala introduces the repair_metadata() function for Iceberg tables. This function provides a self-service recovery path to recover Iceberg tables that are inaccessible due to missing data files after manual file deletions in the underlying storage.

For more information, see Table repair feature.

Apache Jira: IMPALA-13725

Impala Iceberg LIKE predicate pushdown

Impala pushes eligible SQL LIKE predicates on Iceberg tables to Iceberg startsWith() and equal() expressions during query planning. This enables Iceberg to use manifest statistics for file and partition pruning, which can reduce I/O on filtered string scans.

For more information, see LIKE predicate pushdown feature.

Apache Jira: IMPALA-14737

Cloudera Lakehouse Optimizer

Task prioritization feature support

You can define a priority level for maintenance tasks that are configured for a policy. Higher-priority tasks are scheduled before lower-priority tasks when multiple policies compete for cluster resources.

For more information, see Cloudera Lakehouse Optimizer features.

High availability feature support

Cloudera Lakehouse Optimizer supports high availability with multiple service instances, automatic leader election, and failover. At any time, one instance is the leader that accepts API requests and performs state-mutating operations. Standby instances take over automatically when the leader fails.

For more information, see High availability for Cloudera Lakehouse Optimizer.

Puffin statistics feature support

You can create a custom policy that computes NDV statistics for Iceberg table columns and writes them to a Puffin statistics file registered in the table metadata. Query engines that read Iceberg Puffin files use these estimates for cost-based optimization.

For more information, see Puffin statistics for Cloudera Lakehouse Optimizer.

Auditing feature support

Cloudera Lakehouse Optimizer provides an audit trail of user and system activities, including policy creation, editing, deletion, association, namespace operations, authentication events, and table maintenance actions.

For more information, see Auditing for Cloudera Lakehouse Optimizer.

Cloudera Runtime 7.3.2

Cloudera Runtime 7.3.2 introduces new features of Iceberg and includes all service packs and cumulative hotfixes from 7.3.1.100 through 7.3.1.706. For a comprehensive record of all updates in Cloudera Runtime 7.3.1.x, see New Features .

Integrate Iceberg scan metrics into Impala query profiles

Iceberg scan metrics are now integrated into the Frontend section of Impala query profiles, providing deeper insight into query planning performance for Iceberg tables.

The query profile now displays scan metrics from Iceberg's planFiles() API, including total planning time, counts of data/delete files and manifests, and the number of skipped files.

Metrics are displayed on a per-table basis. If a query scans multiple Iceberg tables, a separate metrics section will appear in the profile for each one.

Apache Jira: IMPALA-13628

Delete orphan files for Iceberg tables

You can now use the following syntax to remove orphan files for Iceberg tables:

-- Remove orphan files older than '2022-01-04 10:00:00'.
ALTER TABLE ice_tbl EXECUTE remove_orphan_files('2022-01-04 10:00:00');
            
-- Remove orphan files older than 5 days from now.
ALTER TABLE ice_tbl EXECUTE remove_orphan_files(now() - interval 5 days);
          

This feature removes all files from a table’s data directory that are not linked from metadata files and that are older than the value of older_than parameter. Deleting orphan files from time to time is recommended to keep the size of a table’s data directory under control.

Apache Jira: IMPALA-14492

Allow forced predicate pushdown to Iceberg

Since IMPALA-11591, Impala has optimized query planning by avoiding predicate pushdown to Iceberg unless it is strictly necessary. While this default behavior makes planning faster, it can miss opportunities to prune files early based on Iceberg's file-level statistics.

A new table property, impala.iceberg.push_down_hint is introduced, which allows you to force predicate pushdown for specific columns. The property accepts a comma-separated list of column names, for example, 'col_a, col_b'.

If a query contains a predicate on any column listed in this property, Impala will push that predicate down to Iceberg for evaluation during the planning phase.

Apache Jira: IMPALA-14123

UPDATE operations now skip rows that already have the desired value

The UPDATE statement for Iceberg and Kudu tables is optimized to reduce unnecessary writes.

Previously, an UPDATE operation would modify all rows matching the WHERE clause, even if those rows already contained the new value. For Iceberg tables, this resulted in writing unnecessary new data and delete records.

With this enhancement, Impala automatically adds an extra predicate to the UPDATE statement to exclude rows that already match the target value.

Apache Jira: IMPALA-12588