Fixed Issues in Cloudera Runtime 7.2.18.200

You can review the list of reported issues and their fixes in Cloudera Runtime 7.2.18.200.

OPSAPS-70908: COD Ephemeral Cache ZDU: Refresh cluster command fails due to Auth issue

During refresh command, configurations from refreshable files encountered authentication failure when Kerberos was enabled. This issue is now resolved and Kerberos principal in region server refresh process is now set. RegionServerRefreshCommand now sets SCM_KERBEROS_PRINCIPAL in the environment of the command process since 1-off shell is being created for the same.

OPSAPS-70852, COMPX-17162: [cm 7.12.0.200] Revert feature COMPX-16355 (cm OPSAPS-70184)Fisma compliance changes due to upgrade failures

Reverted FISMA compliance changes due to upgrade failures for Cloudera Manager 7.12.0.200.

OPSAPS-70419: The Livy3 server lacks necessary Iceberg configurations in spark-defaults.

Now Livy3 has all the required Iceberg dependencies similar to Spark3.

OPSAPS-70417: [mow-int] Upgrade failed with unable to start role of Livy service

Added upgrade handler for Livy to set Transport Layer Security (TLS) trust store configuration during the upgrade.

OPSAPS-70335: Some DMP metrics not forwarded when Kerberos is on

If a metrics endpoint scraped by OpenTelemetry Collector requires SPNEGO authentication, it failed if the URL pointed to the localhost. This issue is now resolved.

OPSAPS-70328: Make certain configurations refreshable

The following configurations can now be dynamically configured:

• hbase.rs.evictblocksonclose
• hbase.rs.cacheblocksonwrite
• hbase.block.data.cacheonread

Configuring them dynamically aids in better throughput.

OPSAPS-70297: [dr] Optional Run As User for HBase initial snapshot export

A new option is now added to HBase replication, {{exportSnapshotUser}}. The option is valid only for on-prem to cloud HBase replications where initial snapshots are enabled. When the customer specifies this option when creating the HBase replication policy, the user specified by the new option is used to export the initial snapshot to the target cloud bucket.

OPSAPS-70198: Cloudera Manager Solr to provide zookeeper_znode config value in solr-env.sh

solr-env.sh is now populated with zookeeper-znode configuration.

OPSAPS-69495: Support for secure ZooKeeper connection for Ranger Plugin Solr auditing from Cloudera Manager

The Ranger plugin Solr audit connection configuration is now updated to use a secure port when ZooKeeper is in Secure Sockets Layer (SSL) mode.This fix is provided in Ranger plugin-supported services which are implemented in both Java and CSD framework in Cloudera Manager.

OPSAPS-69336: Cloudera Manager side changes for supporting ability to refresh certain dynamic configuration

Added reference in the hbase-site.xml file to a refreshable configuration file that contains the dynamic configuration.

CDPD-71294: PARQUET-2498 Hadoop vector IO API doesn't handle empty list of ranges

Empty ranges were rejected in Hadoop vector IO and triggred a failure in some tests. This issue is now resolved.

CDPD-71255: Backport IMPALA-12580 to 7.2.18.200

Previously, predicates were not pushed down to Impala scanners if they were already applied by Iceberg and no further rows were filtered. This issue is now resolved and a subset of the predicates are now pushed down to Impala Scan nodes.

CDPD-71008: Backport HBASE-28500 Rest Java client library assumes stateless servers

The Rest Java client library accepts a list of rest servers, and performs random load balancing between them for each request. This did not work for scans, with no state on the rest server instance. This issue is now resolved.

CDPD-71007: Backport HBASE-28526 hbase-rest client shading conflict with hbase-shaded-client in HBase 2.x

There was a shading conflict between hbase-rest client and hbase-shaded client in HBase 2.x. This issue is now reoslved.

CDPD-71006: Backport HBASE-28501 Support non-SPNEGO authentication methods and implement session handling in REST java client library

Added support for non-SPNEGO authentication methods and implemented session handling in REST Java client library.

CDPD-70493: Backport HBASE-28626 MultiRowRangeFilter deserialization fails in org.apache.hadoop.hbase.rest.model.ScannerModel

Previously, the MultiRowRangeFilter deserialization failed in org.apache.hadoop.hbase.rest.model.ScannerModel. This issue is now resolved.

CDPD-70416: Backport HBASE-28613 Use streaming when marshalling protobuf REST output

Previously, protobuf was marshalled into a byte array, and then sent to a client. This was slow and memory intensive. Streaming is now used when marshalling protobuf REST output.

CDPD-70415: Backport HBASE-28556 Reduce memory copying in Rest server when serializing CellModel to Protobuf

The REST server performed unnecessary coping. This issue is now resolved and the memory copying in Rest server when serializing CellModel to Protobuf is now reduced.

CDPD-70155: Zookeeper SSL support for trino

Added ZooKeeper SSL support fro Trino.

CDPD-70004: IMPALA-12681 Some local file descriptors not released when using remote spilling

Fixed an issue where partially written temporary files were removed without releasing the file descriptors.

CDPD-69905: DAS - Upgrade commons-codec to 1.15 or higher

Upgraded the Commons-Codec version to 1.15 and higher.

CDPD-69701, CDPD-69347: UI : If deleted entity has long name, propertytab in UI is misaligned

Previously when an entity was deleted, the property tab of the entity was misaligned. This did not occur when the entity was ACTIVE. This issue is now resolved.

CDPD-69607: Fix for "CDPD-67823 - Ranger RMS gives all permissions to the user through the Create permission" may cause NPE

Ranger RMS gave all permissions to the user throughCreatepermission. This caused an Null Point Exception (NPE) if the ownerUser value for Hive entities in the resource-mappings was not populated. This issue is now resolved.

CDPD-69488: Handle Upgrade failure due to NPE in PatchForUpdatingServiceDefJson_J10058

Fixed an upgrade error failure due to a Null Point Exception (NPE) in PatchForUpdatingServiceDefJson_J10058.

CDPD-69356: Trino: Enable Ranger audit persistence to AWS S3 with HDFS

Trino audit persistence worked with Solr persistence only. Ranger audit persistance to AWS S3 is now wnabled for Trino through HDFS.

CDPD-69335: Backport HBASE-28523 Use a single get call in REST multiget endpoint

The REST multiget endpoint issued a separate HBase GET operation for each key. A new method that accepts a list of keys is now implemented making the process faster.

CDPD-69333: PARQUET-2171: Support Hadoop vectored IO -final merged PR

Added a new feature called Vectored IO in Hadoop for improving read performance for seek heavy readers.

CDPD-69271: Ranger override policy is not working

The override policy in Ranger was not working and the user was denied access. This issue is now resolved.

CDPD-69253: ClientUtilsTest fails because IP addresses changed 7.2.18.x

A unit test in ClientUtilsTest, tests the IP address. It failed if there was a change in the IP addresses. This issue is now resolved.

CDPD-69216: SolrClient support truststore type in ZkClientConfig

Previously, ZkClientConfig supported only truststore path and password. Now, it supports the truststore type.

CDPD-69211: Raz - Zookeeper connection on 2182 port is failing

The Ranger Raz connection with ZooKeeper failed on 2182 port. This issue is now resolved.

CDPD-69154: Update Azure ARM Api version to 2021-03-01

There was an issue due to custom disk encryption policy. This issue is now resolved and the API version is now updated.

CDPD-69051: Ranger - Upgrade Bouncy Castle to 1.78 due to CVE-2024-29857, CVE-2024-30171 and CVE-2024-30172

Upgraded Bouncy Castle version to 1.78 due to CVE-2024-29857, CVE-2024-30171 and CVE-2024-30172.

CDPD-68900: Make properties dynamically configured

The following configurations can now be dynamically configured:

• hbase.rs.evictblocksonclose
• hbase.rs.cacheblocksonwrite
• hbase.block.data.cacheonread

After changing values of these configurations, there is no need to restart the region servers. Hence, such configurations aid in better throughput. Newly changed values in the hbase-site.xml file are read by HBase and values in appropriate classes are updated.

CDPD-68853: [Ranger Trino] Create function and Drop function commands are not supported when Ranger plugin is enabled

When the Ranger Trino plugin was enabled, the Create function and Drop function commands were not supported, and an error message was displayed in the output. This issue is now resolved.

CDPD-68827: [Ranger Trino] Alter materialized view command is not working when Ranger plugin is enabled

When Iceberg catalog was used along with the Ranger plugin enabled for Trino server, the Alter materialized view {view_name} command did not work, and access was denied. This issue is now reoslved.

CDPD-68826: [Ranger Trino] Refresh materialized view command is not working when Ranger plugin is enabled

When Iceberg catalog was used along with the Ranger plugin enabled for Trino server, the Refresh materialized view {view_name} command did not work, and access was denied. This issue is now reoslved.

CDPD-68796: Zeppelin - Upgrade Apache Maven to 3.8.6 due to CVE-2021-26291

Upgraded the Apache Maven version to 3.8.6 to resolve CVE-2021-26291. Now, HTTP (non-SSL) repository references in Project Object Model (POM) files are no longer followed, thereby mitigating the risks of malicious code injection.

CDPD-68692: Output from Hue shows NULL whereas Beeline works

There was an issue where output from a table appeared as NULL when querying from Hue and it happens only for the following quer. This issue is now resolved.

CDPD-68676: The getTopicContent does not always return messages when available

When an individual poll request took a long time to respond, then getTopicContent did not return all messages till the specified end offset. This issue is now resolved. Also, the timeout for the whole getTopicContent request defined in responseTimeOutInMs still applies.

CDPD-68642: MAPREDUCE-7474 [ABFS] Improve commit resilience and performance in Manifest Committer

Improved the commit resilience and performance in the Manifest Committer.

CDPD-68518: Upgrade graal-sdk to 21.3.10 due to CVE-2023-22006 and CVE-2024-21068

Upgraded graal-sdk version to 21.3.10 due to CVE-2023-22006 and CVE-2024-21068.

CDPD-68489: Ranger - Upgrade jline to 3.25.1 due to CVE-2023-50572

Upgraded JLine version to 3.25.1 due to CVE-2023-50572.

CDPD-68434: HADOOP-19141. Vector IO: Update default values consistently

Updated the Vector IO default values.

CDPD-68363: Backporting IMPALA-12798 to CDH-7.2.18.x branch for CR-7.2.18.100 version

Upgraded PostgreSQL version to 42.5.6 due to CVE-2024-1597.

CDPD-68335: Ranger Plugin support to use Solr ZKClientConfig for writing audits to Solr when ZK SSL is enabled

Added ZooKeeper Secure Sockets Layer (SSL) support to Ranger plugin while using audit to Solr.

CDPD-68332: [Ranger Trino] Deleted policies are still taking effect if all policies in a repo are deleted

If all the policies for a security zone were deleted, then an error is seen in the logswhile syncing the policies, and the previously existing policies still took effect and operations were allowed through those policies. This issue is now resolved and nw operations are not allowed through the deleted policies.

CDPD-68278: HWC - Upgrade Netty to 4.1.108.Final due to CVE-2024-29025

Upgraded Netty version to 4.1.108.Final due to CVE-2024-29025.

CDPD-68258: [Ranger Trino] Impersonate access type may not be required for trino policies other than trinouser resource type

The Impersonate access type was being listed in Trino resource based policies such as catalog, schema, table. The Impersonate access type is required for Trino policies when there is the Trinouser resource type. Hence, it is removed.

CDPD-68245: [Ranger trino] Default policies created for cm_trino for policies without select access type cannot be edited without adding permission for rangerlookup user

Policies did not contain the select access type (based on the resource in the policies) in some of the default policies created for cm_trino. When a user tried to edit and save such a policy, then the policy save was not successful as the user was prompted to add an access type for the rangerlookup user. This issue is now resolved and for policies where select access type is not supported, a proper access type is configured for a user.

CDPD-68238: [Ranger Trino] Update operations are not supported when Ranger plugin is enabled

When Ranger Trino plugin was enabled, update operations was authorised, even when the user had all the policies present on all required resources. This issue is now resolved.

CDPD-68178: [Ranger Trino] Audits are not logged for schema/table creation

On a cluster where Trino server was setup and Ranger Trino plugin was enabled, audits were not generated for schema/table creation. This issue is now resolved.

CDPD-67752: [Atlas : 7.2.18.x] - Export/Import : changeMarker is not set to entity's lastupdatetime or its closer timestamp value

When a Hive table entity was exported using a fetch type incremental with changeMarker 0, after exporting, the changeMarker in the export response was not set to a recent timestamp. This issue is now resolved, and the changeMarker is now set to a closer timestamp value during an export or import.

CDPD-67501: Gerrit build failed at cdpd-master-staging

Gerrit build failed at the cdpd-master-staging stage. This issue is now resolved.

CDPD-67338: Handle the ClassCastException of CDPD-40874 in the HWC layer

Previously, the ClassCastException was handled in the Spark layer. This change broke the binary compatibility with stock Spark. This issue is now resolved and it is now handled in the Hive Warehouse Connector (HWC) layer.

CDPD-67336: Revert the Spark change done as part of CDPD-40874, to add Identifier field

Fixed the binary incompatibility issue with stock Spark, so that application code that runs with stock Spark, continues to run seamlessly with CDP Spark distribution.

CDPD-67222: Knox - Upgrade Spring Framework to 6.1.6/6.0.19/5.3.34 due to CVE-2024-22243, CVE-2024-22259 and CVE-2024-22262

UpgradeD Spring Framework version to 6.1.6/6.0.19/5.3.34 due to multiple CVEs.

CDPD-66786: Impala's Iceberg V2 operator produces incorrect results

There was an issue in the PARTITIONED mode when the Iceberg V2 operator processed probe batches that contained rows from multiple data files, and some data files did not have the corresponding delete records. This issue is now resolved and the delete state of the Iceberg V2 operator is reset when records from files do not have delete records.

CDPD-66673: Atlas is not committing messages to Kafka ATLAS_HOOK

Fixed a Null Pointer Exception (NPE) for already processed entities for concurrent ingest performance improvement in Kafka.

CDPD-66298: IMPALA-12788 HBaseTable still get loaded even if HBase is down

Previously, queries were run on HBase tables even when a table was not loaded correctly. The connection failure to HBase was ignored. This issue is now resolved.

CDPD-65373: HBase side changes for making delay prefetch property to be dynamically configured

Rolling restart triggered region movement on a cluster while the RegionServers were restarted. And, the temporary RegionServers started prefetching files that were only hosted until the source RegionServer is restarted. Hence, in this timing window, fetches were executed on temporary region servers which took a few minutes. This issue is now resolved and HBase side changes for making delay prefetch property can now be dynamically configured.

CDPD-64474: Data Catalog Profilers - Upgrade logback to 1.2.13/1.3.14/1.4.14 due to CVE-2023-6378 and CVE-2023-6481

Upgraded Logback to version 1.2.13/1.3.14/1.4.14 due to CVE-2023-6378 and CVE-2023-6481.

CDPD-64216: Spark Schema Registry for Spark 3

Apache Spark 3 is now integrated with Schema Registry. It is a library to leverage Schema Registry for managing Spark schemas and to serialize/de-serialize messages in Spark data sources and sinks.

CDPD-62164: Ranger backup should support different buckets

Ranger backup previously supported only one bucket. It now supports multiple buckets.

CDPD-56444: Add support for branches and tags for iceberg table

Added support for branches and tags for Iceberg tables.

CDPD-55422: Data Catalog Profilers - Upgrade json-smart to 2.4.10 due to CVE-2023-1370

Upgraded JSON-Smart version to 2.4.10 due to CVE-2023-1370.

CDPD-49556: IMPALA-11921 test_large_sql seems to be flaky

There failure in an ASAN run where running test_large_sql resulted in an error. This issue is now resolved.