Fixed issues in 7.1.9 SP1 CHF 5

Know more about the cumulative hotfix 5 for 7.1.9 SP1. This cumulative hotfix was released on 20 Feb, 2025.

Following are the list of fixes that were shipped for CDP Private Cloud Base version 7.1.9-1.cdh7.1.9.p1032.62597146.

CDPD-76634: RMS server throws ConcurrentModificationException

The original ConcurrentModificationException is likely thrown when the resource-mappings are modified in response to changes in the Hive metadata while they are being serialized for downloading to the NameNode (or secondary-namenode).

The fix is to create a shallow copy of resource-mappings before applying deltas which ensures that resource-mappings are not modified while they are being serialized for downloading to the NameNode.

CDPD-75981: Default native ACL limits to user and user's primary group

Default native ACL created for an object such as volume, bucket, or file limits to the object owner and owner's primary group.

Apache Jira: HDDS-11656

CDPD-77877: Exclude guava from arrow-vector

Fixed an issue where a transitive dependency would force custom applications to be built with a particular version of guava.

CDPD-59896: Hue fails to clean up documents if Oozie is on the blocked list of services

Earlier, when you ran the desktop_document_cleanup script to clean old documents, and if you added Oozie to the blocked list of apps in the Hue Advanced Configuration Snippet, the script failed. To run the script successfully, you had to remove the Oozie service from the app_blacklist property manually. This issue has been resolved. You no longer need to manually remove Oozie from the blocked list of apps.

CDPD-62397: Improved default editor selection in Hue configuration

Previously, when you tried to log in and access Hue, the first available editor was used by default if multiple editors were configured. This issue has now been resolved, allowing you to specify a default editor in the Hue configuration file for all users.

CDPD-75286: Spark History UI - StreamConstraintsException: String length exceeds the maximum length

Fixed an issue with Jackson to allow unlimited json string length in Spark event logs.

Zeppelin Livy interpreter paragraph outputs are being misplaced

Fixed an issue where Zeppelin paragraph outputs were being misplaced. Users are now able to use pyspark to run os.system commands without issues.

CDPD-78260: Partition value mismatch during INSERT event processing

Impala failed to process some INSERT events due to partition value mismatches. Hive encodes partition strings in file paths using URL encoding, but partition strings in HMS events are not encoded. Impala mismatched partitions during HMS events processing by decoding them incorrectly, causing event processing failures.

The fix includes decoding partition strings only when derived from file paths and not from HMS events to ensure accurate partition mapping.

Apache Jira: IMPALA-13691

CDPD-77922: Prometheus metric name incompatibility with JDK11 and JDK17

Impala metrics derived from JVM MemoryPoolMXBean names contain apostrophe characters in JDK11 and JDK17. Since Prometheus does not support apostrophes in metric names, these metrics cannot be consumed by Prometheus.

To address the issue, apostrophes in Prometheus metric names are now automatically translated to underscores, ensuring compatibility with Prometheus requirements.

Apache Jira: IMPALA-13638

CDPD-73427: Upgrade Spring Framework to resolve vulnerabilities

CVE-2024-38808: Vulnerable to resource exhaustion through user-supplied SpEL expressions.

CVE-2024-38809: DoS attack possible due to improper ETag validation in HTTP requests.

Upgraded Spring Framework to versions 6.1.12, 6.0.23, or 5.3.39 to mitigate these issues.

CDPD-75665: Importing a table generates a DDL with an incorrect location

When creating a table using the IMPORT command, an external imported table can have this partitions located under the managed warehouse directory, which violates the metastore.warehouse.external.dir and metastore.warehouse.dir that intend to host different type of tables.

The fix ensures that the table path is fetched from HMS before importing partitions, ensuring that all data is copied to the correct location in the newly imported table.

Apache Jira: HIVE-28580

CDPD-74658: Invalid plan generated for IN subquery with multiple correlations

Hive was generating incorrect query plans for IN subqueries with multiple correlations, causing errors due to missing column references.

This issue is now fixed.

Apache Jira: HIVE-24999

CDPD-76983 CDPD-76982: Removed redundant code

This issue is now fixed.

Apache Jira: HIVE-24412

Apache Jira: HIVE-18725

CDPD-76985: Optimize scalar query rewrite to avoid unnecessary joins

Hive planner adds an unnecessary join for scalar subqueries that return at most one row.

The subquery removal rule now checks row count estimates and skips adding the extra join.

Apache Jira: HIVE-21537

CDPD-76980: Support EXISTS/NOT EXISTS correlated subqueries with aggregates

Hive failed to handle correlated EXISTS/NOT EXISTS subqueries containing aggregate functions without a GROUP BY clause, leading to query compilation errors.

This functionality is now supported.

Apache Jira: HIVE-15757

CDPD-76978: Improve error handling for invalid column references in subqueries

Hive returned a misleading error when a subquery referenced a non-existing column, making it unclear that the issue was a missing column

Removed HiveSubQRemoveRelBuilder and used RelBuilder.

Apache Jira: HIVE-24685

CDPD-74657: Incorrect results when using COALESCE in correlated subqueries

Queries with a correlated subquery using the COALESCE function in the correlation predicate returned incorrect results.

The query execution logic was updated to correctly apply COALESCE in correlated subqueries, ensuring that all matching rows are included in the result set.

Apache Jira: HIVE-24957

CDPD-76984: Incorrect query results due to merging TableScan operators with different Dynamic Partition Pruning parents

The shared work optimizer merged TableScan operators that had different Dynamic Partition Pruning (DPP) parents. This caused missing filters in the query plan, leading to incorrect results.

The issue is now fixed.

Apache Jira: HIVE-26968

CDPD-78077: Kafka Connect connectors might fail during deployment or while being moved to another node

Kafka Connect connectors no longer fail due to incompletely unpacked dependencies while being deployed or moved to another node.

CDPD-76135: Schema check database connection throws PSQLException

The schema check now completes successfully during Oozie startup for secure database connections or when custom connection properties are used, preventing warnings from appearing in the logs.

CDPD-76604: CDEP fails to install CDP Private Cloud Base as Ozone does not start up. The OZONE-1 has failed in startup so CDEP hasn't executed next set of commands. error is displayed.

This issue is resolved by fixing the race condition in creation and deletion of rocksdbjni native library.

COMPX-18909: NodeManager can be marked unhealthy if an application is killed

A logic was introduced to mark the NodeManager unhealthy if the container-executor encountered a unrecoverable/configuration error, that is, when the container-executor script was missing. If the application was killed just before when one of the containers tried to access the localizer syslog file an IOException occured due to the deleted app log directory, and then the NodeManager was marked unhealthy incorrectly.

This issue is now resolved and the error checking is more specific and the false positive NodeManager markings are now removed.

Apache Jira: YARN-11753

CDPD-49702: NodeManager must be shut down when the program /var/lib/yarn-ce/bin/container-executor cannot be run

Previoulsy, a job failed when NodeManager encountered the No such file or directory error when running the /var/lib/yarn-ce/bin/container-executor program. This issue is now resolved and NodeManager is marked as unhealthy and shut down when it cannot run the program.

Apache Jira: YARN-11709

CDPD-74510: Migration status is not updated on file while migration is in progress

Previously, whenever a migration was started on any environment and tried to update the vertex with details, the status update would fail with a null pointer exception. This issue is fixed by using the file hash code instead of file name causing the null pointer exception. Now, whenever a migration restarts, it starts from last point where it was stopped or interrupted.