Fixed Issues in Apache Hive

Review the list of Hive issues that are resolved in Cloudera Runtime 7.1.9.

CDPD-27233: ACID HMS table names should changed to VARCHAR(256)

In the Hive metastore schema, there were some tables where the column width for certain columns was inconsistent across tables. This has been addressed.

CDPD-34727: Graceful shutdown of HiveServer2

With the introduction of a graceful shutdown state for HiveServer (HS2), HS2 will wait for running (in-flight) queries to complete within a configured amount of time before shutting down.

CDPD-39708: Use cdpd_guava_version from external_versions.ini

This patch uses guava dependency version from cdpd repro.

CDPD-43493: Upgrade JUnit to 4.13.2 due to medium CVEs

Upgraded JUnit to 4.13.2 to fix CVEs.

CDPD-43540: Upgrade jersey to 2.36/3.0.5 due to medium CVEs

Upgraded jersey to 2.36 to fix CVEs.

CDPD-45017: Backport HIVE-23242: Fix flaky tests testHouseKeepingThreadExistence in TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader

Re-enable the HMS leader test in TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader.

CDPD-47132: Pushdown Date data type to Hive metastore via direct sql / JDO

Fix partition filtering when querying partition metadata from metastore and partition key column data type is date.

CDPD-47464: Alter view command allowed even when user has a deny policy on the underlying table

"Alter View As" queries were not authorized correctly. This fix addresses the security concern around the authorization of "Alter View As" queries.

CDPD-47557: SparklyRHWC certification with R4

SparklyrHWC supports R version 4.0.5.

CDPD-48022: Upgrade postgresql to 42.5.1 due to CVE-2022-41946

Upgraded PostgreSQL to 42.5.1 to fix CVEs

CDPD-48801: Pushdown Timestamp data type to Hive metastore via direct sql / JDO

Support partition filtering when querying partition metadata from metastore and partition key column data type is timestamp.

CDPD-48989: Hive/Impala tests in CDH-7.1.8.x branch fails with DataNucleus connection timeout errors

This fix introduces a secondary connection pool for HMS DataNucleus value generator to avoid connection starvation.

CDPD-49145: Oozie and Spark tests are failing in multi-comp-pre with Zookeeper-based or direct JDBC URL to Hive

HiveServer (HS2) uses InetAddress.getHostName() API to get its hostname and register itself with Zookeeper. The API behaviour is changed on JDK 11 with specific operating systems to return only the hostname without the domain suffix. Therefore, HS2 is inaccessible to clients when the server information is obtained from Zookeeper. To address this, the InetAddress.getCanonicalHostName() API is used to fetch the hostname along with the fully qualified domain name.

CDPD-49492: Extend batch partition APIs to ignore partition schemas

This patch addresses issues with high partition workloads. See the documentation for information about the recommended configurations required to run high partition workloads.

CDPD-49507: {OWNER} policy not working with HIVE UDFs in RangerHiveAuthorizer

The UDFs used in Hive will now honor {Owner} polices in Ranger with this fix.

CDPD-50148: Add double quotes for tables in PartitionProjectionEvaluator

This fix addresses an issue where a missing relation PSQLException is returned when PartitionProjectionEvaluator requests partitions against PostgreSQL.

CDPD-50450: Backport HIVE-27201: Inconsistency between session Hive and thread-local Hive may cause HS2 deadlock

Two HiveServer (HS2) sessions can go into a deadlock state due to RANGER-3593 and can indefinitely wait for each other. This fix resolves the deadlock condition.

CDPD-50464: Installation failures with Hive errors

The fix sets a different name for the connection pool for Compactor to resolve conflict with the connection metrics of pool for ObjectStore.

CDPD-50730: Hive WebUI HTTP 500 error due to jar order in classpath

Removed javax.servlet.jsp-api dependency from HiveServer (HS2) to avoid the intermittent Null Pointer Exception while opening the home web page.

CDPD-51885: Fix and backport HIVE-27163 - Column stats are not getting published after an insert query into an external table with custom location

Column stats are published after an insert into the external table created with an empty custom location.

CDPD-53363: Backport HIVE-25032: Optimize PartitionManagementTask

Optimize PartitionManagementTask by searching for the required tables to be repaired and dropping the obsolete partitions in a bulk.

CDPD-55511: CDPD-39232 / RANGER-3593 causing slow execution of SHOW TABLEs command

Currently, TableMeta does not include ownership information which makes it difficult for filterTableMetas to efficiently filter based on ${OWNER} privileges.

Ranger will look up ownership information for database objects (databases/tables/views) through HMS api calls during authorization checks, if it is not provided by the callee. This can be very slow for commands like SHOW TABLEs when the database has a large amount of tables, since it may generate a call per object.

This fix addresses the issue by providing ownership information for these commands from the Hive side so that the Ranger authorization plugin is no longer required to make HMS calls in this situation.

CDPD-55866: Backport HIVE-18827 to CDH-7.1.9.x

Fixed unneccessary dynamic value exceptions

CDPD-55867: Backport HIVE-23295 to CDH-7.1.9.x

This fix addresses a potential Null Pointer Exception when dynamic values are not available while fetching the predicate literal list.

CDPD-55868: Backport HIVE-23410 to CDH-7.1.9.x

As part of an earlier fix, insert operation was modified to write directly to the table location instead of the staging directory. The same improvement is now available for ACID update and delete operations.

CDPD-55869: Backport HIVE-24581 to CDH-7.1.9.x

The split generation in OrcInputformat is tightly coupled with ACID and AcidUtils.getAcidState is called even when the table is not transactional. The fix includes the following changes:

• Removes unneccessary AcidUtils.getAcidState call from OrcInputformat when the table is not transactional
• Removes redundant filesystem utility functions from AcidUtils to HdfsUtils

Removed AcidUtils call from OrcInputformat for non transactional tables

CDPD-55870: Backport HIVE-24669 to CDH-7.1.9.x

Improved FileSystem usage in Hive::loadPartitionInternal to improve performance.

CDPD-55871: Backport HIVE-24679 to CDH-7.1.9.x

In order to improve performance, FullDPSpecs in loadDynamicPartitions are reused to avoid double listing.

CDPD-55873: Backport HIVE-24682 to CDH-7.1.9.x

The dynamic partition information is collected from the FileSink for direct insert and reused later in the MoveTask.

CDPD-55875: Backport CDPD-36395 to CDH-7.1.9.x

Spark/Spark3 builds fail due to Hive changes for CDPD-6264/HIVE-23410

CDPD-58160: Backport CDPD-56782 to 7.1.9

This fix addresses an issue where DIRECT_READER_V2 mode returned wrong values for string columns after merge query.

CDPD-59419: Backport CDPD-59091 to 7.1.9

Return getDefaultSession() if the Spark getActiveSession() returns None.

OPSAPS-66334: Make modifying hive.security.authorization.sqlstd.confwhitelist.append as first party config

This fix introduces the hive.security.authorization.sqlstd.confwhitelist.append configuration in Cloudera Manager. Users can use this configuration to add Hive properties that are failing insttead of the earlier approach of adding this in "Hive Client Advanced Configuration Snippet (Safety Valve) for hive-site.xml".

OPSAPS-64733: Adding Hive with Ozone warehouse directory fails

The Hive warehouse directory and external warehouse directory can be set to Ozone during the first Hive run. The Ozone filesystem JAR is available as part of the hdfs.sh script (in the /opt/cloudera/cm/lib/cdh7 directory).

OPSAPS-67942: Installation failed due to schematool error

Setting the hive.hook.proto.base-directory for Hive Metastore (HMS) in hive-site.xml is causing sys.db creation to fail because of incompatibility issues between Cloudera Manager 7.11.3 and CDH 7.1.7 SP1/SP2. This patch addresses the issue and sets the above configuration only if the CDH version of Hive is at least CDH 7.1.8.

OPSAPS-68074: CDH7.1.7SP2 Rolling upgrade to 7.1.9 fails to validate Validating metastore schema tables

Added upgrade handler for Hive for CDP 7.1.9 release.

OPSAPS-68213: NFQE JDK17 runtime: module java.base does not issue (java.util.regex, java.io, java.time,java.nio)

This fix addresses an issue where java.base does not issue (java.util.regex, java.io, java.time,java.nio).