What's New in Apache Iceberg

Learn about the new features of Iceberg in Cloudera Runtime 7.3.2, its service packs and cumulative hotfixes.

Cloudera Runtime 7.3.2.10000 SP1

Upgrade Hive Iceberg library to Apache Iceberg 1.10.0

In Cloudera Runtime 7.3.2 SP1, the Apache Iceberg library used by the Hive engine is upgraded to Apache Iceberg 1.10.0. Existing Iceberg V1 and V2 table operations are unchanged; you need not run a new SQL syntax for current workflows.

This upgrade aligns Hive with the platform-wide Iceberg 1.10.0 library and provides the foundation for Iceberg V3 capabilities in Hive. For supported Iceberg features, see the Apache Iceberg Feature Support Matrix.

Apache Jira: HIVE-28665

Upgrade Impala Iceberg library to Apache Iceberg 1.10.0

In Cloudera Runtime 7.3.2 SP1, the Apache Iceberg library used by the Impala engine is upgraded to Apache Iceberg 1.10.0. Existing Iceberg V1 and V2 table operations are unchanged; you need not run a new SQL syntax for current workflows.

This upgrade aligns Impala with the platform-wide Iceberg 1.10.0 library and provides the foundation for Iceberg V3 capabilities in Impala. For supported Iceberg features, see the Apache Iceberg Feature Support Matrix.

Apache Jira: IMPALA-14665

Hive Iceberg variant data type support

Hive supports the Iceberg VARIANT data type. The Apache Iceberg format specification defines this semi-structured type for Iceberg format version 3. You can store semi-structured data in VARIANT columns and query nested fields using PARSE_JSON and VARIANT_GET (HIVE-29183).

For more information, see Iceberg data types and Create table feature.

Apache Jira: HIVE-29183

Hive Iceberg deletion vector support

Hive supports deletion vectors on Iceberg V3 tables. Deletion vectors are the Iceberg V3 encoding for position deletes and replaces the V2 position delete files for new delete operations on V3 tables. When you run DELETE on a V3 Iceberg table, Hive records deleted row positions using deletion vectors.

For more information, see Delete data feature and Row-level operations.

Apache Jira: HIVE-29006

Impala Iceberg deletion vector support [Technical Preview]

Impala supports deletion vectors on Iceberg V3 tables. Deletion vectors are the Iceberg V3 encoding for position deletes and replace V2 position delete files for new delete operations on V3 tables. Impala can read tables that contain deletion vectors written by Impala or by other Iceberg V3-compatible engines. When you run DELETE on a V3 Iceberg table from Impala, Impala records deleted row positions using deletion vectors.

For more information, see Delete data feature and Row-level operations.

Apache Jira: IMPALA-14585

Impala Iceberg row lineage support [Technical Preview]

Impala supports row lineage tracking on Iceberg format V3 tables. Row lineage is an Iceberg V3 capability that assigns a unique identifier to each row and records row provenance across table snapshots. This metadata supports change data capture (CDC) use cases. To use row lineage from Impala, create an Iceberg V3 table by setting 'format-version'='3'. When Impala writes to the table, Impala maintains the row lineage metadata required; you need not run a new SQL syntax to enable row lineage.

For more information, see Row lineage feature.

Apache Jira: IMPALA-14592

Impala Iceberg puffin statistics support

Impala supports writing Puffin statistics files for Iceberg V3 tables. Puffin is a lightweight container format that stores binary metadata such as deletion vector sidecar files required by the Iceberg V3 specification.

For more information, see Puffin statistics feature.

Apache Jira: IMPALA-14755

Impala Iceberg table repair support

Impala introduces the repair_metadata() function for Iceberg tables. This function provides a self-service recovery path to recover Iceberg tables that are inaccessible due to missing data files after manual file deletions in the underlying storage.

For more information, see Table repair feature.

Apache Jira: IMPALA-13725

Impala Iceberg LIKE predicate pushdown

Impala pushes eligible SQL LIKE predicates on Iceberg tables to Iceberg startsWith() and equal() expressions during query planning. This enables Iceberg to use manifest statistics for file and partition pruning, which can reduce I/O on filtered string scans.

For more information, see LIKE predicate pushdown feature.

Apache Jira: IMPALA-14737

Cloudera Lakehouse Optimizer

Cloudera Lakehouse Optimizer UI support for Cloudera Base on premises: On Cloudera Base on premises, you can deploy the Lakehouse Optimizer UI parcel in Cloudera Manager, add the Lakehouse UI service to your cluster, and configure the Knox Gateway URL to access the Cloudera Lakehouse Optimizer UI.
For more information, see Deploying and configuring Cloudera Lakehouse Optimizer UI.
Task prioritization feature support: You can define a priority level for maintenance tasks that are configured for a policy. Higher-priority tasks are scheduled before lower-priority tasks when multiple policies compete for cluster resources.
For more information, see Cloudera Lakehouse Optimizer features.
High availability feature support: Cloudera Lakehouse Optimizer supports high availability with multiple service instances, automatic leader election, and failover. At any time, one instance is the leader that accepts API requests and performs state-mutating operations. Standby instances take over automatically when the leader fails.
For more information, see High availability for Cloudera Lakehouse Optimizer.
Puffin statistics feature support: You can create a custom policy that computes NDV statistics for Iceberg table columns and writes them to a Puffin statistics file registered in the table metadata. Query engines that read Iceberg Puffin files use these estimates for cost-based optimization.
For more information, see Puffin statistics for Cloudera Lakehouse Optimizer.
Auditing feature support: Cloudera Lakehouse Optimizer provides an audit trail of user and system activities, including policy creation, editing, deletion, association, namespace operations, authentication events, and table maintenance actions.
For more information, see Auditing for Cloudera Lakehouse Optimizer.
State persistence feature support (Cloudera Base on premises only): On Cloudera Base on premises, Cloudera Lakehouse Optimizer persists scheduler and task state including dispatched tasks, event-based schedules, and in-flight Livy task submission data in a shared database so maintenance work witholds restarts and high availability failover.
For more information, see State persistence for Cloudera Lakehouse Optimizer.
Data file compression feature support (Cloudera Base on premises only): On Cloudera Base on premises, Cloudera Lakehouse Optimizer can re-encode Iceberg data files that match a SQL filter with a more efficient compression codec to reduce storage costs without changing query results.
For more information, see Data file compression for Cloudera Lakehouse Optimizer.

Cloudera Runtime 7.3.2

Cloudera Runtime 7.3.2 introduces new features of Iceberg and includes all service packs and cumulative hotfixes from 7.3.1.100 through 7.3.1.706. For a comprehensive record of all updates in Cloudera Runtime 7.3.1.x, see New Features.

Cloudera Lakehouse Optimizer for Iceberg table optimization

In Cloudera Runtime 7.3.2 and higher versions, you can use Cloudera Lakehouse Optimizer service in Cloudera Manager to automate the Iceberg table maintenance tasks.

Cloudera Lakehouse Optimizer provides automated Iceberg table maintenance, through Spark jobs, for Iceberg tables in Cloudera Open Data Lakehouse. It simplifies table management, improves query performance, and reduces operational costs.

You can add the service to an existing Cloudera Base on premises 7.3.2 or higher versions cluster in Cloudera Manager 7.13.2 or higher versions, or you can create a dedicated cluster and then add the service. You must ensure that the cluster contains all the required services. After you finish configuring the service, you can use the Cloudera Lakehouse Optimizer service REST APIs to define the Cloudera Lakehouse Optimizer policies, perform policy management, and run other Iceberg table optimization operations.

For more information, see Cloudera Lakehouse Optimizer.

Integrate Iceberg scan metrics into Impala query profiles

Iceberg scan metrics are now integrated into the Frontend section of Impala query profiles, providing deeper insight into query planning performance for Iceberg tables.

The query profile now displays scan metrics from Iceberg's planFiles() API, including total planning time, counts of data/delete files and manifests, and the number of skipped files.

Metrics are displayed on a per-table basis. If a query scans multiple Iceberg tables, a separate metrics section will appear in the profile for each one.

Apache Jira: IMPALA-13628

Delete orphan files for Iceberg tables

You can now use the following syntax to remove orphan files for Iceberg tables:

-- Remove orphan files older than '2022-01-04 10:00:00'.
ALTER TABLE ice_tbl EXECUTE remove_orphan_files('2022-01-04 10:00:00');
            
-- Remove orphan files older than 5 days from now.
ALTER TABLE ice_tbl EXECUTE remove_orphan_files(now() - interval 5 days);

This feature removes all files from a table’s data directory that are not linked from metadata files and that are older than the value of older_than parameter. Deleting orphan files from time to time is recommended to keep the size of a table’s data directory under control.

Apache Jira: IMPALA-14492

Allow forced predicate pushdown to Iceberg

Since IMPALA-11591, Impala has optimized query planning by avoiding predicate pushdown to Iceberg unless it is strictly necessary. While this default behavior makes planning faster, it can miss opportunities to prune files early based on Iceberg's file-level statistics.

A new table property, impala.iceberg.push_down_hint is introduced, which allows you to force predicate pushdown for specific columns. The property accepts a comma-separated list of column names, for example, 'col_a, col_b'.

If a query contains a predicate on any column listed in this property, Impala will push that predicate down to Iceberg for evaluation during the planning phase.

Apache Jira: IMPALA-14123

UPDATE operations now skip rows that already have the desired value

The UPDATE statement for Iceberg and Kudu tables is optimized to reduce unnecessary writes.

Previously, an UPDATE operation would modify all rows matching the WHERE clause, even if those rows already contained the new value. For Iceberg tables, this resulted in writing unnecessary new data and delete records.

With this enhancement, Impala automatically adds an extra predicate to the UPDATE statement to exclude rows that already match the target value.

Apache Jira: IMPALA-12588