Known Issues in Apache Hive

Known issues identified in Cloudera Runtime 7.3.2.100 CHF 1

There are no new known issues identified in this release.

Known issues identified in Cloudera Runtime 7.3.2

CDPD-98415: Hive ACID incremental replication fails with "MetaException: invalid write id"

7.3.2.0

During Hive ACID incremental replication, the replication load task fails with a metastore exception similar to the following:

Caused by: org.apache.hadoop.hive.metastore.api.MetaException: invalid write id <ID> for table <db_name>.<table_name>

This issue occurs when processing a large volume of incremental replication events (such as bulk DDLs, partition operations, or TRUNCATE TABLE queries).

When the load builder batches events across multiple task iterations (controlled by hive.repl.approx.max.load.tasks), proactive transaction cleanup logic (CLEAR_DANGLING_TXNS) executes prematurely at the end of a batch iteration instead of at the end of the full incremental load operation. If an open transaction is replayed in one batch, but its corresponding commit event is scheduled in the subsequent batch, the cleanup task prematurely aborts the in-flight target transaction and invalidates its write ID.

If an upgrade to Cloudera Runtime 7.1.9 SP2 is not immediately possible, disable the proactive dangling transaction cleanup logic and enforce strict database cleanup ordering:

1. Disable dangling transaction cleanup:

   Set hive.repl.clear.dangling.txns.on.target to false globally in Cloudera Manager under Hive Service Advanced Configuration Snippet (Safety Valve) or explicitly in both REPL DUMP and REPL LOAD commands:

   1. Source Cluster (REPL DUMP):

      REPL DUMP <db_name> WITH (
        'hive.repl.replica.external.table.base.dir'='hdfs://<target-hdfs-host>:8020/',
        'hive.repl.rootdir'='hdfs://<target-hdfs-host>:8020/warehouse/tablespace/staging/<db_name>',
        'hive.repl.run.data.copy.tasks.on.target'='true',
        'hive.repl.clear.dangling.txns.on.target'='false'
      );

   2. Target Cluster (REPL LOAD):

      REPL LOAD <db_name> INTO <db_name> WITH (
        'hive.repl.replica.external.table.base.dir'='hdfs://<target-hdfs-host>:8020/',
        'hive.repl.rootdir'='hdfs://<target-hdfs-host>:8020/warehouse/tablespace/staging/<db_name>',
        'hive.repl.run.data.copy.tasks.on.target'='true',
        'hive.repl.clear.dangling.txns.on.target'='false'
      );

2. Enforce database cleanup order during re-bootstrap:

   When hive.repl.clear.dangling.txns.on.target is set to false, administrators must follow a strict order during re-bootstrap operations to prevent orphan entries in the Hive Metastore:

   1. Delete the Replication Policy First: Always remove the replication schedule or policy before deleting the database.

   2. Drop the Target Database Second: Dropping the database before the policy removes database metadata while leaving stale entries in REPL_TXN_MAP.

   3. Verify Metastore State: Before initiating a re-bootstrap load, run the following SQL query directly on the Target Hive Metastore database to verify no lingering entries exist for the policy:

      SELECT * FROM REPL_TXN_MAP WHERE RTM_REPL_POLICY = '<db_name>.*';

Apache Jira: HIVE-29459

CDPD-73781: High operational costs from automatic partition discovery on cloud storage

7.3.1 and its SPs and CHFs, 7.2.18 and its SPs

In cloud environments, the Partition Management Task (PMT) could significantly increase operational costs and performance overhead for large tables with many partitions.

To address this issue, you can perform the following:

• Increase the value of the metastore.partition.management.task.frequency property to 86400 seconds.
• Disable partition discovery manually for specific tables incurring high costs by using the discover.partitions table property.

CDPD-93090: Failure when inserting data into partitions located on a different filesystem than the table

7.3.2

Inserting data into a table partition results in a MoveTask execution error if the partition is stored in a different filesystem than the parent table. For example, if you create an external table in S3 but add a partition with an HDFS location, the INSERT operation fails with a Wrong FS error. Although the data is physically inserted into the correct location, the query returns a non-zero exit code and displays an error message.

None

CDPD-92560: Partition insertion errors across multiple buckets in RAZ enabled clusters

7.3.2

Inserting data into a table partition results in a MoveTask execution error when the partition is located in a different S3 bucket than the base table. In Ranger Access Control (RAZ) enabled Private Cloud clusters, the INSERT operation fails with a Wrong FS error, even though the data is physically written to the specified partition location.

None

CDPD-74192: Hive SSL certificate error during large record copies

7.3.1 and its SPs and CHFs, 7.2.18.300, 7.2.18.400

When you copy a large number of records (approximately 1 million or more) using Hive, the operation might fail with a

javax.net.ssl.SSLException: org.bouncycastle.tls.TlsFatalAlert: certificate_unknown(46)

error. This issue occurs during the job commit phase when Hive attempts to execute HTTP requests to S3 for file operations such as copyFile.

None

CDPD-68096: Unable to set S3 credentials at the session level in Hive

7.3.2

Accessing S3 buckets from Hive on on premises results in a MetaException or AccessDeniedException when you attempt to create external tables by using session-level credentials. While Spark and other Hadoop jobs allow you to define fs.s3a.access.key and fs.s3a.secret.key at the session level, Hive Metastore (HMS) does not honor these session-specific parameters during path validation.

To access S3 buckets, you must configure the S3 credentials at the global level in the Hive or Hadoop cluster configuration (such as core-site.xml) or use a supported credential provider like the Hadoop Credential Shell to store keys securely. Setting these parameters at the session level is currently not supported for Hive table creation.

HIVE-16913

Known issues identified before Cloudera Runtime 7.3.2

DWX-22436: DL upgrade recovery fails due to Metastore schema incompatibility

7.3.1.600

When attempting a Data Lake (DL) upgrade recovery from version 7.2.18.1100 to Cloudera Runtime 7.3.1.500, the process fails because the Hive Metastore schema versions are incompatible. The error indicates a mismatch between the Hive version (3.1.3000.7.3.1.500-182) and the database schema version (3.1.3000.7.2.18.0-Update2). This blocks Data Lake recovery if an upgrade fails, impacting customers.

Before you initiate the recovery process, manually update the Hive Metastore schema to match the target version by using the schematool utility.

1. Obtain the Hive database password: Run the following command to retrieve the password from the pillar configuration:

   cat /srv/pillar/postgresql/postgre.sls

2. Back up the existing configuration: Move the current configuration directory to a backup location:

   mv /etc/hive/conf /etc/hive/conf_backup
                     mkdir /etc/hive/conf

3. Prepare the temporary configuration: Copy the process files to the new configuration directory:

   scp /var/run/cloudera-scm-agent/process/<process-id>-hive-metastore-create-tables/* /etc/hive/conf/

4. Update the connection password: Open the /etc/hive/conf/hive-site.xml file and perform the following modifications:
   • Set the javax.jdo.option.ConnectionPassword property to your Hive database password.
   • Comment out the hadoop.security.credential.provider.path property.
5. Run the schema upgrade tool: Execute the schematool to synchronize the version:

   /opt/cloudera/parcels/CDH/lib/hive/bin/schematool -dbType postgres -initOrUpgradeSchema --verbose

6. Restore the original configuration: Remove the temporary directory and restore your backup:

   rm -rf /etc/hive/conf
                     mv /etc/hive/conf_backup /etc/hive/conf

7. Restart the cluster: Restart the services to initialize the Hive Metastore with the updated schema.

CDPD-77738: Atlas hook authorization issue causing HiveCreateSysDb timeout

7.1.9 SP1 CHF4, 7.1.7 SP3 CHF7, 7.3.1.100, and its higher versions

Atlas hook authorization error causes HiveCreateSysDb command to time out due to repeated retries.

None

CDPD-74680: DAG not retried after failure

7.3.1 and its higher versions

When executing a Hive query, if the ApplicationMaster container fails, Hive does not retry the DAG if the failure message contains some diagnostic information including a line break, leading to query failure (instead of retry).

None