Fixed Issues in Cloudera Runtime 7.2.18.200
You can review the list of reported issues and their fixes in Cloudera Runtime 7.2.18.200.
- OPSAPS-70908: COD Ephemeral Cache ZDU: Refresh cluster command fails due to Auth issue
- During refresh command, configurations from refreshable files encountered
authentication failure when Kerberos was enabled. This issue is now resolved and Kerberos
principal in region server refresh process is now set.
RegionServerRefreshCommand now sets
SCM_KERBEROS_PRINCIPAL
in the environment of the command process since 1-off shell is being created for the same. - OPSAPS-70852, COMPX-17162: [cm 7.12.0.200] Revert feature COMPX-16355 (cm OPSAPS-70184)Fisma compliance changes due to upgrade failures
- Reverted FISMA compliance changes due to upgrade failures for Cloudera Manager 7.12.0.200.
- OPSAPS-70419: The Livy3 server lacks necessary Iceberg configurations in spark-defaults.
- Now Livy3 has all the required Iceberg dependencies similar to Spark3.
- OPSAPS-70417: [mow-int] Upgrade failed with unable to start role of Livy service
- Added upgrade handler for Livy to set Transport Layer Security (TLS) trust store configuration during the upgrade.
- OPSAPS-70335: Some DMP metrics not forwarded when Kerberos is on
- If a metrics endpoint scraped by OpenTelemetry Collector requires SPNEGO authentication, it failed if the URL pointed to the localhost. This issue is now resolved.
- OPSAPS-70328: Make certain configurations refreshable
- The following configurations can now be dynamically configured:
hbase.rs.evictblocksonclose
hbase.rs.cacheblocksonwrite
hbase.block.data.cacheonread
- OPSAPS-70297: [dr] Optional Run As User for HBase initial snapshot export
- A new option is now added to HBase replication,
{{exportSnapshotUser}}
. The option is valid only for on-prem to cloud HBase replications where initial snapshots are enabled. When the customer specifies this option when creating the HBase replication policy, the user specified by the new option is used to export the initial snapshot to the target cloud bucket. - OPSAPS-70198: Cloudera Manager Solr to provide zookeeper_znode config value in solr-env.sh
- solr-env.sh is now populated with zookeeper-znode configuration.
- OPSAPS-69495: Support for secure ZooKeeper connection for Ranger Plugin Solr auditing from Cloudera Manager
- The Ranger plugin Solr audit connection configuration is now updated to use a secure port when ZooKeeper is in Secure Sockets Layer (SSL) mode.This fix is provided in Ranger plugin-supported services which are implemented in both Java and CSD framework in Cloudera Manager.
- OPSAPS-69336: Cloudera Manager side changes for supporting ability to refresh certain dynamic configuration
- Added reference in the hbase-site.xml file to a refreshable configuration file that contains the dynamic configuration.
- CDPD-71294: PARQUET-2498 Hadoop vector IO API doesn't handle empty list of ranges
- Empty ranges were rejected in Hadoop vector IO and triggred a failure in some tests. This issue is now resolved.
- CDPD-71255: Backport IMPALA-12580 to 7.2.18.200
- Previously, predicates were not pushed down to Impala scanners if they were already applied by Iceberg and no further rows were filtered. This issue is now resolved and a subset of the predicates are now pushed down to Impala Scan nodes.
- CDPD-71008: Backport HBASE-28500 Rest Java client library assumes stateless servers
- The Rest Java client library accepts a list of rest servers, and performs random load balancing between them for each request. This did not work for scans, with no state on the rest server instance. This issue is now resolved.
- CDPD-71007: Backport HBASE-28526
hbase-rest
client shading conflict with hbase-shaded-client in HBase 2.x - There was a shading conflict between
hbase-rest
client andhbase-shaded
client in HBase 2.x. This issue is now reoslved. - CDPD-71006: Backport HBASE-28501 Support non-SPNEGO authentication methods and implement session handling in REST java client library
- Added support for non-SPNEGO authentication methods and implemented session handling in REST Java client library.
- CDPD-70493: Backport HBASE-28626 MultiRowRangeFilter deserialization fails in org.apache.hadoop.hbase.rest.model.ScannerModel
- Previously, the
MultiRowRangeFilter
deserialization failed inorg.apache.hadoop.hbase.rest.model.ScannerModel
. This issue is now resolved. - CDPD-70416: Backport HBASE-28613 Use streaming when marshalling protobuf REST output
- Previously, protobuf was marshalled into a byte array, and then sent to a client. This was slow and memory intensive. Streaming is now used when marshalling protobuf REST output.
- CDPD-70415: Backport HBASE-28556 Reduce memory copying in Rest server when serializing CellModel to Protobuf
- The REST server performed unnecessary coping. This issue is now resolved and the memory copying in Rest server when serializing CellModel to Protobuf is now reduced.
- CDPD-70155: Zookeeper SSL support for trino
- Added ZooKeeper SSL support fro Trino.
- CDPD-70004: IMPALA-12681 Some local file descriptors not released when using remote spilling
- Fixed an issue where partially written temporary files were removed without releasing the file descriptors.
- CDPD-69905: DAS - Upgrade commons-codec to 1.15 or higher
- Upgraded the Commons-Codec version to 1.15 and higher.
- CDPD-69701, CDPD-69347: UI : If deleted entity has long name, propertytab in UI is misaligned
- Previously when an entity was deleted, the property tab of the entity was misaligned. This did not occur when the entity was ACTIVE. This issue is now resolved.
- CDPD-69607: Fix for "CDPD-67823 - Ranger RMS gives all permissions to the user through the Create permission" may cause NPE
- Ranger RMS gave all permissions to the user
through
Create
permission. This caused an Null Point Exception (NPE) if the ownerUser value for Hive entities in the resource-mappings was not populated. This issue is now resolved. - CDPD-69488: Handle Upgrade failure due to NPE in PatchForUpdatingServiceDefJson_J10058
- Fixed an upgrade error failure due to a Null Point Exception (NPE) in
PatchForUpdatingServiceDefJson_J10058
. - CDPD-69356: Trino: Enable Ranger audit persistence to AWS S3 with HDFS
- Trino audit persistence worked with Solr persistence only. Ranger audit persistance to AWS S3 is now wnabled for Trino through HDFS.
- CDPD-69335: Backport HBASE-28523 Use a single get call in REST multiget endpoint
- The
REST
multiget endpoint issued a separate HBaseGET
operation for each key. A new method that accepts a list of keys is now implemented making the process faster. - CDPD-69333: PARQUET-2171: Support Hadoop vectored IO -final merged PR
- Added a new feature called Vectored IO in Hadoop for improving read performance for seek heavy readers.
- CDPD-69271: Ranger override policy is not working
- The override policy in Ranger was not working and the user was denied access. This issue is now resolved.
- CDPD-69253: ClientUtilsTest fails because IP addresses changed 7.2.18.x
- A unit test in
ClientUtilsTest
, tests the IP address. It failed if there was a change in the IP addresses. This issue is now resolved. - CDPD-69216: SolrClient support truststore type in ZkClientConfig
- Previously,
ZkClientConfig
supported only truststore path and password. Now, it supports the truststore type. - CDPD-69211: Raz - Zookeeper connection on 2182 port is failing
- The Ranger Raz connection with ZooKeeper failed on 2182 port. This issue is now resolved.
- CDPD-69154: Update Azure ARM Api version to 2021-03-01
- There was an issue due to custom disk encryption policy. This issue is now resolved and the API version is now updated.
- CDPD-69051: Ranger - Upgrade Bouncy Castle to 1.78 due to CVE-2024-29857, CVE-2024-30171 and CVE-2024-30172
- Upgraded Bouncy Castle version to 1.78 due to CVE-2024-29857, CVE-2024-30171 and CVE-2024-30172.
- CDPD-68900: Make properties dynamically configured
- The following configurations can now be dynamically configured:
hbase.rs.evictblocksonclose
hbase.rs.cacheblocksonwrite
hbase.block.data.cacheonread
- CDPD-68853: [Ranger Trino] Create function and Drop function commands are not supported when Ranger plugin is enabled
- When the Ranger Trino plugin was enabled, the Create function and Drop function commands were not supported, and an error message was displayed in the output. This issue is now resolved.
- CDPD-68827: [Ranger Trino] Alter materialized view command is not working when Ranger plugin is enabled
- When Iceberg catalog was used along with the Ranger plugin enabled for Trino server, the Alter materialized view {view_name} command did not work, and access was denied. This issue is now reoslved.
- CDPD-68826: [Ranger Trino] Refresh materialized view command is not working when Ranger plugin is enabled
- When Iceberg catalog was used along with the Ranger plugin enabled for Trino server, the Refresh materialized view {view_name} command did not work, and access was denied. This issue is now reoslved.
- CDPD-68796: Zeppelin - Upgrade Apache Maven to 3.8.6 due to CVE-2021-26291
- Upgraded the Apache Maven version to 3.8.6 to resolve CVE-2021-26291. Now, HTTP (non-SSL) repository references in Project Object Model (POM) files are no longer followed, thereby mitigating the risks of malicious code injection.
- CDPD-68692: Output from Hue shows NULL whereas Beeline works
- There was an issue where output from a table appeared as NULL when querying from Hue and it happens only for the following quer. This issue is now resolved.
- CDPD-68676: The getTopicContent does not always return messages when available
- When an individual poll request took a long time to respond, then
getTopicContent
did not return all messages till the specified end offset. This issue is now resolved. Also, the timeout for the wholegetTopicContent
request defined inresponseTimeOutInMs
still applies. - CDPD-68642: MAPREDUCE-7474 [ABFS] Improve commit resilience and performance in Manifest Committer
- Improved the commit resilience and performance in the Manifest Committer.
- CDPD-68518: Upgrade graal-sdk to 21.3.10 due to CVE-2023-22006 and CVE-2024-21068
- Upgraded graal-sdk version to 21.3.10 due to CVE-2023-22006 and CVE-2024-21068.
- CDPD-68489: Ranger - Upgrade jline to 3.25.1 due to CVE-2023-50572
- Upgraded JLine version to 3.25.1 due to CVE-2023-50572.
- CDPD-68434: HADOOP-19141. Vector IO: Update default values consistently
- Updated the Vector IO default values.
- CDPD-68363: Backporting IMPALA-12798 to CDH-7.2.18.x branch for CR-7.2.18.100 version
- Upgraded PostgreSQL version to 42.5.6 due to CVE-2024-1597.
- CDPD-68335: Ranger Plugin support to use Solr ZKClientConfig for writing audits to Solr when ZK SSL is enabled
- Added ZooKeeper Secure Sockets Layer (SSL) support to Ranger plugin while using audit to Solr.
- CDPD-68332: [Ranger Trino] Deleted policies are still taking effect if all policies in a repo are deleted
- If all the policies for a security zone were deleted, then an error is seen in the logswhile syncing the policies, and the previously existing policies still took effect and operations were allowed through those policies. This issue is now resolved and nw operations are not allowed through the deleted policies.
- CDPD-68278: HWC - Upgrade Netty to 4.1.108.Final due to CVE-2024-29025
- Upgraded Netty version to 4.1.108.Final due to CVE-2024-29025.
- CDPD-68258: [Ranger Trino] Impersonate access type may not be required for trino policies other than trinouser resource type
-
The
Impersonate
access type was being listed in Trino resource based policies such as catalog, schema, table. TheImpersonate
access type is required for Trino policies when there is theTrinouser
resource type. Hence, it is removed. - CDPD-68245: [Ranger trino] Default policies created for cm_trino for policies without select access type cannot be edited without adding permission for rangerlookup user
- Policies did not contain the select access type (based on the
resource in the policies) in some of the default policies created for
cm_trino
. When a user tried to edit and save such a policy, then the policy save was not successful as the user was prompted to add an access type for the rangerlookup user. This issue is now resolved and for policies where select access type is not supported, a proper access type is configured for a user. - CDPD-68238: [Ranger Trino] Update operations are not supported when Ranger plugin is enabled
- When Ranger Trino plugin was enabled, update operations was authorised, even when the user had all the policies present on all required resources. This issue is now resolved.
- CDPD-68178: [Ranger Trino] Audits are not logged for schema/table creation
- On a cluster where Trino server was setup and Ranger Trino plugin was enabled, audits were not generated for schema/table creation. This issue is now resolved.
- CDPD-67752: [Atlas : 7.2.18.x] - Export/Import : changeMarker is not set to entity's lastupdatetime or its closer timestamp value
- When a Hive table entity was exported using a fetch type incremental with changeMarker 0, after exporting, the changeMarker in the export response was not set to a recent timestamp. This issue is now resolved, and the changeMarker is now set to a closer timestamp value during an export or import.
- CDPD-67501: Gerrit build failed at cdpd-master-staging
- Gerrit build failed at the cdpd-master-staging stage. This issue is now resolved.
- CDPD-67338: Handle the ClassCastException of CDPD-40874 in the HWC layer
- Previously, the ClassCastException was handled in the Spark layer. This change broke the binary compatibility with stock Spark. This issue is now resolved and it is now handled in the Hive Warehouse Connector (HWC) layer.
- CDPD-67336: Revert the Spark change done as part of CDPD-40874, to add Identifier field
- Fixed the binary incompatibility issue with stock Spark, so that application code that runs with stock Spark, continues to run seamlessly with CDP Spark distribution.
- CDPD-67222: Knox - Upgrade Spring Framework to 6.1.6/6.0.19/5.3.34 due to CVE-2024-22243, CVE-2024-22259 and CVE-2024-22262
- UpgradeD Spring Framework version to 6.1.6/6.0.19/5.3.34 due to multiple CVEs.
- CDPD-66786: Impala's Iceberg V2 operator produces incorrect results
- There was an issue in the PARTITIONED mode when the Iceberg V2 operator processed probe batches that contained rows from multiple data files, and some data files did not have the corresponding delete records. This issue is now resolved and the delete state of the Iceberg V2 operator is reset when records from files do not have delete records.
- CDPD-66673: Atlas is not committing messages to Kafka ATLAS_HOOK
- Fixed a Null Pointer Exception (NPE) for already processed entities for concurrent ingest performance improvement in Kafka.
- CDPD-66298: IMPALA-12788 HBaseTable still get loaded even if HBase is down
- Previously, queries were run on HBase tables even when a table was not loaded correctly. The connection failure to HBase was ignored. This issue is now resolved.
- CDPD-65373: HBase side changes for making delay prefetch property to be dynamically configured
- Rolling restart triggered region movement on a cluster while the RegionServers were restarted. And, the temporary RegionServers started prefetching files that were only hosted until the source RegionServer is restarted. Hence, in this timing window, fetches were executed on temporary region servers which took a few minutes. This issue is now resolved and HBase side changes for making delay prefetch property can now be dynamically configured.
- CDPD-64474: Data Catalog Profilers - Upgrade logback to 1.2.13/1.3.14/1.4.14 due to CVE-2023-6378 and CVE-2023-6481
- Upgraded Logback to version 1.2.13/1.3.14/1.4.14 due to CVE-2023-6378 and CVE-2023-6481.
- CDPD-64216: Spark Schema Registry for Spark 3
- Apache Spark 3 is now integrated with Schema Registry. It is a library to leverage Schema Registry for managing Spark schemas and to serialize/de-serialize messages in Spark data sources and sinks.
- CDPD-62164: Ranger backup should support different buckets
- Ranger backup previously supported only one bucket. It now supports multiple buckets.
- CDPD-56444: Add support for branches and tags for iceberg table
- Added support for branches and tags for Iceberg tables.
- CDPD-55422: Data Catalog Profilers - Upgrade json-smart to 2.4.10 due to CVE-2023-1370
- Upgraded JSON-Smart version to 2.4.10 due to CVE-2023-1370.
- CDPD-49556: IMPALA-11921 test_large_sql seems to be flaky
- There failure in an ASAN run where running
test_large_sql
resulted in an error. This issue is now resolved.