Known Issues in Apache Iceberg

Known issues and technical limitations for Iceberg are addressed in Cloudera Runtime 7.3.3.

Known Issues identified in Cloudera Runtime 7.3.3

CDPD-127140: Hive fails on COUNT(DISTINCT ROW__LINEAGE__ID) on Iceberg format version 3 tables
7.3.3
On Iceberg format version 3 tables, a plain SELECT of row lineage metadata columns such as ROW__LINEAGE__ID and LAST__UPDATED__SEQUENCE__NUMBER succeeds in Hive. When vectorized execution is enabled by default, a query such as SELECT COUNT(DISTINCT ROW__LINEAGE__ID) FROM on the table fails during the vectorized Iceberg scan while reading Parquet data. Hive returns an IndexOutOfBoundsException from the vectorized Parquet record reader. The same distinct-count check on row lineage columns in Impala and Spark completes successfully.
Disable vectorized execution for the session or query by setting hive.vectorized.execution.enabled=false or avoid COUNT(DISTINCT ROW__LINEAGE__ID).
CDPD-127281: Hive applies FOR SYSTEM_VERSION AS OF to every scan of the same Iceberg table in one query
7.3.3, 7.3.2
In Hive, when a single statement scans the same Iceberg table more than once and only one scan uses FOR SYSTEM_VERSION AS OF, Hive can apply the snapshot version to all scans of that table in the statement. The scan that should read the current table then returns the same row set as the time-travel scan. Separate queries that read only the current table or only a snapshot returns correct results. Impala and Spark return correct counts when the current table and a snapshot are compared in one query.
Compare a snapshot to the current table by running separate queries. For example, one SELECT count(*) with FOR SYSTEM_VERSION AS OF and one without instead of combining both scans of the same Iceberg table in one statement.

Apache Jira: HIVE-27190

CDPD-127289: Hive does not expose ROW__LINEAGE__ID on Iceberg format version 3 ORC or Avro tables
7.3.3
On Iceberg format version 3 tables in Hive, ROW__LINEAGE__ID is the metadata column for the Iceberg row ID. When write.format.default is a parquet file format, Hive accepts SELECT references to ROW__LINEAGE__ID. When the table uses ORC or Avro data files, Hive fails at compile time with an SemanticException [Error 10004] invalid column reference ROW__LINEAGE__ID and lists only user-defined table columns. Row lineage metadata is supported only for Parquet file format in Hive in this release.
To query ROW__LINEAGE__ID or LAST__UPDATED__SEQUENCE__NUMBER from Hive, create or migrate the Iceberg format version 3 table with write.format.default=parquet in table properties.
CDPD-127285: Impala returns duplicate _row_id values on Iceberg format version 3 ORC and Avro tables after an update
7.3.3
On Iceberg format version 3 tables, Impala exposes _row_id as row lineage metadata. After a large UPDATE on tables that use ORC or Avro data files, Impala can return duplicate _row_id values across rows. A check such as count(DISTINCT _row_id) = count(*) then returns false even though every row has a non-null _row_id. The same workflow on format version 3 Parquet tables return distinct IDs for every row. Impala reads ORC and Avro Iceberg tables but does not write those formats in the result scenario.
For Iceberg format version 3 tables that require stable, use write.format.default=parquet for unique row IDs in Impala after updates.

Known issues identified before Cloudera Runtime 7.3.3

CDPD-111552: Unsupported Iceberg column type alterations in Hive return unhandled IllegalArgumentException
7.3.2.20000
Using Hive to change the data type of a column on an Iceberg table to an unsupported conversion, such as BOOLEAN to STRING or a narrowing change such as BIGINT to INT results in the statement failure with an unhandled IllegalArgumentException and a Java stack trace. Hive does not return a concise, user-facing error message (for example, that the conversion is not supported). Supported Iceberg type promotions, such as INT to BIGINT, complete successfully. The same behavior occurs in Beeline.
Use only Iceberg-supported column type promotions when you alter Iceberg table schemas. Unsupported conversions are rejected. If a statement fails, use the Cannot change column type message in the error output to identify the unsupported conversion.
CDPD-101742: Hive Iceberg snapshot expiration fails for ORC LZ4/LZO tables
7.3.2.10000
Hive fails to run queries on Iceberg tables stored in ORC format with LZ4 or LZO compression when LLAP cache is enabled. The failure results in an OWN_TASK_FAILURE status and vertex failures due to an UnsupportedOperationException during ORC decompression. The snapshot expiration functionality itself works as expected and is not impacted.
For Iceberg tables, disable vectorization by setting hive.vectorized.execution.enabled=false.
CDPD-89390/CDPD-83022: Incorrect row count displayed in table metadata after compaction
7.3.2.10000, 7.3.2, 7.3.1.600, 7.3.1.500, 7.3.1.400
After running data compaction operations on large tables, the row count displayed by the DESCRIBE FORMATTED command may be inaccurate. Initially, the count may appear higher than the actual number of rows. Subsequently, after running the ANALYZE command to update table statistics, the count might then appear lower than the actual number of rows.

This issue has been observed in large tables containing a significant number of historical snapshots (exceeding 10000). All these snapshots are primarily generated through UPDATE operations.

It is important to note that this is just a metadata display issue, and there is no loss of data. The underlying table data remains complete and correct.

To obtain an accurate row count, use the SELECT COUNT(*) query.
DWX-18843: Unable to read Iceberg table from Hive Virtual Warehouse
7.3.2
If you have used Apache Flink to insert data into an Iceberg table that is created from Hive, you cannot read the Iceberg table from the Hive Virtual Warehouse.
Add the engine.hive.enabled table property through the Hive beeline and set the value to "true". You can add this table property either while creating the Iceberg table or use the ALTER TABLE statement to add the table property.
DWX-18489: Hive compaction of Iceberg tables results in a failure
7.3.2
When Cloudera Data Warehouse and Cloudera Data Hub are deployed in the same environment and use the same Hive Metastore (HMS) instance, the Cloudera Data Hub compaction workers can inadvertently pick up Iceberg compaction tasks. Since Iceberg compaction is not yet supported in the latest Cloudera Data Hub version, the compaction tasks will fail when they are processed by the Cloudera Data Hub compaction workers.

In such a scenario where both Cloudera Data Warehouse and Cloudera Data Hub share the same HMS instance and there is a requirement to run both Hive ACID and Iceberg compaction jobs, it is recommended that you use the Cloudera Data Warehouse environment for these jobs. If you want to run only Hive ACID compaction tasks, you can choose to use either the Cloudera Data Warehouse or Cloudera Data Hub environments.

If you want to run the compaction jobs without changing the environment, it is recommended that you use Cloudera Data Warehouse. To avoid interference from Cloudera Data Hub, change the value of the hive.compactor.worker.threads Hive Server (HS2) property to '0'. This ensures that the compaction jobs are not processed by Cloudera Data Hub.
  1. In Cloudera Manager, click Clusters > Hive > Configuration to navigate to the configuration page for HMS.
  2. Search for hive.compactor.worker.threads and modify the value to '0'.
  3. Save the changes and restart the Hive service.
DWX-17254: Merging Iceberg branches requires a target table alias
7.3.2
Hive supports only one level of qualifier when referencing columns. In other words only one dot is accepted. For example, select table.col from ...; is allowed. select db.table.col is not allowed. Using the merge statement to merge Iceberg branches without a target or source table alias causes an exception:
org.apache.hadoop.hive.ql.parse.SemanticException: ... Invalid table alias or column reference ...
Use an alias, for example t, for the target table.
merge into mydb.target.branch_branch1 t using mydb.source.branch_branch1 s on t.id = s.id when matched then update set value = 'matched';

Apache Jira: HIVE-28055

DWX-17210, DWX-13733: Timeout issue querying Iceberg tables from Hive
7.3.2
When querying Iceberg tables from Hive, the queries can faile due to a timeout issue.
  1. Add the following configurations to hadoop-core-site for the Database Catalog and the Virtual Warehouse.
    • fs.s3.maxConnections=1000
    • fs.s3a.connection.maximum=1000
  2. Restart the Database Catalog and Virtual Warehouse.
DWX-14163: Limitations reading Iceberg tables in Avro file format from Impala
7.3.2
The Avro, Impala, and Iceberg specifications describe some limitations related to Avro, and those limitations exist in Cloudera. In addition to these, the DECIMAL type is not supported in this release.
None.
DEX-7946: Data loss during migration of a Hive table to Iceberg
7.3.2
In this release, by default the table property 'external.table.purge' is set to true, which deletes the table data and metadata if you drop the table during migration from Hive to Iceberg.
Either one of the following workarounds prevents data loss during table migration:
  • Set the table property 'external.table.purge'='FALSE'.
  • Do not drop a table during migration from Hive to Iceberg.
DWX-13062: Converting a Hive table having CHAR or VARCHAR columns to Iceberg causes an exception
7.3.2
CHAR and VARCHAR data can be shorter than the length specified by the data type. Remaining characters are padded with spaces. Data is converted to a string in Iceberg. This process can yield incorrect results when you query the converted Iceberg table.
Change columns from CHAR or VARCHAR to string types before converting the Hive table to Iceberg.

Apache Jira: HIVE-26507

CDPD-92182: Inserting into Hive Iceberg tables on S3 fails with RazS3ClientCredentialsException
7.3.2, 7.3.1.706, 7.3.1.600, 7.3.1.500
In RAZ-enabled clusters where HDFS is the default file system, attempts to insert data into Hive Iceberg tables pointing explicitly to S3 locations fail with a RazS3ClientCredentialsException
None.
CDPD-89390/CDPD-83022: Incorrect row count displayed in table metadata after compaction
7.3.2, 7.3.1.600, 7.3.1.500, 7.3.1.400
After running data compaction operations on large tables, the row count displayed by the DESCRIBE FORMATTED command may be inaccurate. Initially, the count may appear higher than the actual number of rows. Subsequently, after running the ANALYZE command to update table statistics, the count might then appear lower than the actual number of rows.

This issue has been observed in large tables containing a significant number of historical snapshots (exceeding 10000). All these snapshots are primarily generated through UPDATE operations.

It is important to note that this is just a metadata display issue, and there is no loss of data. The underlying table data remains complete and correct.

To obtain an accurate row count, use the SELECT COUNT(*) query.

Cloudera Lakehouse Optimizer

CDPD-119573: Evaluation API returns an error
7.3.2.10000
The evaluation API returns a Duplicate evaluation request error when stale task metadata exists in the submitted state.
Delete the stale task metadata that is in the submitted state for the affected table from storage (HDFS or S3).