Cloudera Docs

Iceberg replication policies

Iceberg replication policies replicate Iceberg V1 and V2 tables stored on HDFS and Ozone between Cloudera Base on premises clusters.

The following table lists the supported Iceberg replication features, minimum version requirements, and compatibility rules:
Feature Minimum supported versions Version compatibility requirement

Replicate Spark-created Iceberg V1 and V2 tables on HDFS (Read-only with Impala)

Cloudera Runtime 7.1.9 using Cloudera Manager 7.11.3

Supports mixed versions (backward compatibility) between source and target clusters.

Replicate Hive-created Iceberg V1 and V2 tables on HDFS

  • Cloudera Manager 7.13.1.x

  • Cloudera Runtime 7.3.1.x
Support for the Cloudera Manager 7.13.1.x and Cloudera Runtime 7.3.1.x release lines depends on the following version-matching conditions:
  • Versions prior to Cloudera Manager 7.13.1.900 / Cloudera Runtime 7.3.1.900

    When source and target clusters run versions lower than Cloudera Manager 7.13.1.900 and Cloudera Runtime 7.3.1.900, Cloudera supports replication only when both clusters run identical Cloudera Manager and Cloudera Runtime versions.

    Cloudera does not support replication across mixed or different versions (for example, a source running Cloudera Manager 7.13.1.200 / Cloudera Runtime 7.3.1.200 and a target running Cloudera Manager 7.13.1.400 / Cloudera Runtime 7.3.1.400).

  • Cloudera Manager 7.13.1.900 or higher / Cloudera Runtime 7.3.1.900 or higher

    Cloudera supports mixed-version compatibility (backward compatibility) between source and target clusters when both clusters run Cloudera Manager 7.13.1.900 or higher and Cloudera Runtime 7.3.1.900 or higher.

Replicate Spark or Hive-created Iceberg V1 and V2 tables stored on Ozone buckets (Spark: read-only with Impala).

  • Cloudera Runtime 7.3.2.0 using Cloudera Manager 7.13.2.0

  • Cloudera Runtime 7.1.9 SP2 using Cloudera Manager 7.13.1.800

Supports mixed versions (backward compatibility) between source and target clusters.
​​Apache Iceberg is a cloud-native, high-performance open table format for organizing petabyte-scale analytic datasets on a file system or object store. Iceberg supports ACID-compliant tables, including row-level deletes and updates and can define large analytic data tables using open format files.
Iceberg replication policies provide the following functionalities:
  • Replicating metadata and catalog from the source cluster Hive Metastore (HMS) to the target cluster HMS.
  • Replicating data files in the HDFS storage system or Ozone storage system from the source cluster to the target cluster. The Iceberg replication policies can replicate only between HDFS storage systems or between Ozone storage systems.
  • Replicating data at table level.
  • Replicating all the snapshots from the source cluster which allows you to run time travel queries on the target cluster.
Some use cases where you can use Iceberg replication policies are to:
  • replicate Iceberg tables between on-premises clusters to archive data or run analytics,
  • implement passive disaster recovery with planned failover and perform incremental replication at regular intervals between two similar systems. For example, between an HDFS to another HDFS system.

This video demonstrates the ability of the Iceberg replication policy to replicate multiple Iceberg tables from diverse locations in the source cluster to a target cluster in a single replication job. It also showcases the Hive on Iceberg feature that allows you to replicate Iceberg tables created by Hive and Impala, and also a use case related to the location mapping feature to map the source path and the target path. These features are available in Cloudera Base on premises 7.3.1 using Cloudera Manager 7.13.1 and higher versions.

Figure 1. Using an Iceberg replication policy to replicate all Iceberg tables located in diverse locations in the source cluster