Iceberg overview

Apache Iceberg is a table format for huge analytics datasets in the cloud that defines how metadata is stored and data files are organized. Iceberg is also a library that compute engines can use to read/write a table.

You can efficiently query large Iceberg tables on object stores. Iceberg supports concurrent reads and writes on all storage media. You can use Iceberg when a single table contains tens of petabytes of data, and you can read these tables without compromising performance. From Apache Hive and Apache Impala, you can read and write V1 or V2 Iceberg tables. The following features are included:

Reads of Iceberg V2 tables that have had row-level deletes or updates makes Apache Iceberg ACID-compliant with serializable isolation and an optimistic concurrency model
Materialized views of Iceberg tables
Enhanced maintenance features, such as expiring and removing old snapshots and compaction of small files
Performance and scalability enhancements

In Cloudera Public Cloud Data Hub, you can deploy Iceberg based applications across multiple clouds including AWS, Azure and Google Cloud.

The Hive metastore stores Iceberg metadata, including the location of the table.

Hive metastore plays a lightweight role in the Catalog operations. Iceberg relieves Hive metastore (HMS) pressure by storing partition information in metadata files on the file system/object store instead of within the HMS. This architecture supports rapid scaling without performance hits.

By default, Hive and Impala use the Iceberg HiveCatalog. Cloudera recommends the default HiveCatalog to create an Iceberg table. HadoopFileIO is supported.

Apache Iceberg integrates Apache Ranger for security. You can use Ranger integration with Hive and Impala to apply fine-grained access control to sensitive data in Iceberg tables. Iceberg is also integrated with Data Visualization for creating dashboards and other graphics of your Iceberg data.