Fixed issues in 7.1.9 SP1 CHF 2

CDPD-64979: CORS is too permissive for the public APIs

This issue has been resolved by adding the following three properties to control Cross-Origin Resource Sharing (CORS) under the [desktop] section in the hue configuration file:

cors_enabled: Used to enable or disable CORS. The default value is True.
cors_allow_credentials: Used to determines whether the server allows cookies in the cross-site HTTP requests. The default value is True.
cors_allowed_origins: A comma-separated list of origins allowed for CORS. For example:
```
[desktop]
cors_allowed_origins=[***ORIGIN-1***],[***ORIGIN-2***],[***ORIGIN-3***]
```
If no values are set, then all origins are allowed.

To change the values of these properties, go to Cloudera Manager > Clusters > Hue > Configuration > Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini, and specify the configuration in the [desktop] section with the required value. For example, to disable CORS, specify the following:

[desktop]
cors_enabled=false

COMPX-17702: Backport - YARN-10345 - HsWebServices containerlogs does not honor ACLs for completed jobs

Previously, the following rest APIs did not have any authorization:

/ws/v1/history/containerlogs/{containerid}/{filename}
/ws/v1/history/containers/{containerid}/logs

This is now resolvedx and the APIs now have an ACL authorization.

CDPD-73943: HIVE-27994: Optimized renaming of partitioned tables

Previously, renaming partitioned tables in Hive required fetching, deleting, and re-inserting every row in PART_COL_STATS. This is now optimized to improve performance, especially for tables with large datasets.

CDPD-73942: HIVE-25225: Fix for NPE in update column stats with disabled direct SQL

In batch loading scenarios, overhead during partition column stat updates is now reduced.

CDPD-73941: HIVE-24663: Optimized partition column stats update for large batches

When large number of partitions (>20K) were processed, the ColStatsProcessor ran into DB issues. This issue is now resolved by introducing smaller batches for stats gathering, preventing bulk update issues.

CDPD-73795: Backport CALCITE-6530 to 7.1.9 SP1 CHF2

Http Sessions created by the Avatica server never expired causing Out of Memory issues. This issue is now resolved.

CDPD-73761: [7.1.9 SP1 CHF2] Backport KAFKA-15391 Delete topic may lead to directory offline

Fixed an issue where deleting a topic caused the entire log directory to go offline.

CDPD-73678: Atlas - [7.1.9 SP1 CHFx]Upgrade Spring Framework to 6.1.12/6.0.23/5.3.39 due to CVE-2024-38808 and CVE-2024-38809

Upgraded the Spring Framework version to 6.1.12 due to CVE-2024-38808 and CVE-2024-38809.

CDPD-73643: Filter out unused CM_USER parameter

The unused CM_USER field is now removed from the /cm-configs internal endpoint.

CDPD-73442: IMPALA-13313 Resolution of potential deadlock

When idle_query_timeout was set in a session, new queries stopped responding for that session and failed. This deadlock issue occurred in long-running sessions and is now resolved.

CDPD-73423: Ranger - Upgrade Spring Framework to 6.1.12/6.0.23/5.3.39 due to CVE-2024-38808 and CVE-2024-38809

Upgraded the Spring-framework version to 5.3.39 due to CVE-2024-38808 and CVE-2024-38809.

CDPD-73326: Reduce memory needed to create Ranger policy engine

An issue led to the creation of multiple RangerResourceMatchers with identical resource specification. This issue is now resolved and the creation of multiple RangerResourceMatcher objects is now avoided by maintaining a cache of them in the RangerPluginContext object associated with the Ranger policy engine, thereby reducing policy engine's memory needs.

CDPD-73282: Backport CALCITE-6530 HTTP Sessions are never expired in Avatica server

The http sessions created by the Avatica server did not expire and this caused the Avatica server to run out of memory. This issue is now resolved.

CDPD-73226: Add /tmp noexec support for Zstd and Snappy compression

Zstandard and Snappy compression now support /tmp mounted as noexec.

CDPD-73147: [Ranger React UI] Admin audits for "Import Delete" operation type do not display service name field

In the Ranger React UI, in admin audits, the Service name field was missing for the audits of operation type Import Delete. This issue is now resolved and the Import delete policy logs now display the service name.

CDPD-73144: Enhance trie to support processing of evaluators during traversal

Ranger policy engine uses trie data structure to organize resources for faster retrieval of policies/tags/zones associated with a given resource. When a resource consists of multiple elements, such as, database/table/column, many trie instances are consulted to retrieve policies/tags/zones associated with the resource.

Such multi-trie retrieval is optimized with a 2-pass traversal - first pass to get count and the second pass to get the actual objects. Thherefore, the trie data structure used in Ranger policy engine is now updated to support processing of evaluators during traversal.

CDPD-72812: SSTFilteringService was disabled by default causing snapshot chain corruption.

SSTFilteringService was disabled by default causing snapshot chain corruption. The Ozone snapshots SSTFilteringService is now enabled by default.

CDPD-72766: CLONE - /api/atlas/admin/version works only once after atlas restart

API, Atlas, Admin, Version now works without Atlas restart.

CDPD-72621: Support for default constraints in HWC table writes

Added functionality in Hive Warehouse Connector to support default constraints during table writes, enhancing data integrity and management.

CDPD-72596: IMPALA-12921 IMPALA-12985: Support for running Impala with locally built Ranger

The fix adds support for locally built Ranger and for using the new constructor when instantiating RangerAccessRequestImpl.

CDPD-72292: [Private Cloud Releases] Upgrade RequireJS due to CVE-2024-38998 and CVE-2024-38999

Upgraded the RequireJS version due to CVE-2024-38998 and CVE-2024-38999.

CDPD-71764: XSS vulnerability in Zeppelin : Unsanitized HTML in Markdown Paragraphs

To enhance security, Zeppelin now integrates HTML sanitization using JSoup within the markdown interpreter. This ensures that any HTML embedded in markdown is sanitized according to a configurable blacklist.

CDPD-71279: Proposal to Upgrade All React.js Dependent Libraries

Upgraded the React.JS related libraries.

CDPD-71063: Hue - Upgrade eventlet to 0.35.2 and dnspython to 2.6.1 due to CVE-2023-29483

Security fix for CVE-2023-29483.

CDPD-69767: Backport fix to address CVE-2023-50291

This ticket backports the fix for CVE-2023-50291 from upstream Solr repository.

CDPD-69634, CDPD-69412: [719 SP1 CHFx] DB constraint violation error for grant and revoke command execution

Multiple column revoke resulted in incorrect number of columns in the generated policies, and the revoke statement only revoked if there was only one column. This issue is now resolved and multiple column revoke now generates correct results.

CDPD-69411: [719 SP1 CHFx] Impala's authorization-related tests failed due to PSQLException due to Ranger's backend database

Impala's authorization-related tests failed due to PSQLException caused by Ranger's backend database. This issue is now resolved.

CDPD-67913: IMPALA-12554 Consolidation of Ranger policies for GRANT statements in Impala

Impala now creates a single Ranger policy for the GRANT statement, even when multiple columns are specified, reducing the overall number of policies on the Ranger server.

CDPD-67552: [7.1.9] Add optional wait time for replication-records-lag calculation

Streams Replication Manager (SRM) added a new metric (replication-records-lag) about replication flows. The calculation of this metric has an impact on the performance of the SRM, so the following 3 properties were introduced to be this feature configurable. These properties can be specified in the Cloudera Manager, Configuration tab of the Streams Replication Manager service, where new lines has to be added to the "streams.replication.manager.config" property, where key-value pairs can be defined. The "replication.records.lag.calc.enabled" flag helps to turn on or off the new feature. The calculation of this metric is enabled by default. The "replication.records.lag.calc.period.ms" provides an opportunity to lower the frequency of the calculation of replication-records-lag. The default value is 0, so the metric will be provided every time, but for example with 15 sec or with longer periods, the calculation will not have a huge effect on the SRM replication latency, but the new metric will be available, so it will be not entirely switched off. The "replication.records.lag.end.offset.timeout.ms" is a way to specify the Kafka end offset timeout value in case of replication-records-lag calculation. The default value is 1 min, but lower value provides the opportunity to reduce latency, beside the risk that replication-records-lag calculation may fail. On the other hand, the higher value helps to avoid the failure of the metric calculation, but it may have bigger effect on the SRM replication latency.

CDPD-67460: Container Balancer must move only containers with size greater than 0 bytes.

Introduced a check on the size of the containers allowed to leave the source node during the balancing process. Added a unit test TestContainerBalancerTask#balancerShouldMoveOnlyPositiveSizeContainers to ensure that a balancer selects only the positive size containers. This is tested using the existing parameter TestContainerBalancerTask#balancerShouldObeyMaxSizeLeavingSourceLimit.

CDPD-67113: [7.1.9] Backport KAFKA-13988: Mirrormaker 2 auto.offset.reset=latest not working

Streams Replication Manager (SRM) did not respond to auto.offset.reset=latest config. This issue is now resolved.

CDPD-66886: Solr - Backport fix for CVE-2023-50298, CVE-2023-50292, CVE-2023-50386 and CVE-2023-50291

This ticket backports the fix for CVE-2023-50298, CVE-2023-50292, CVE-2023-50386 and CVE-2023-50291 from upstream Solr repository.

CDPD-64939: [719 SP1 CHFx] RANGER-4585 Support multiple columns policy creation in ranger for Grant / Revoke request

Upgrading from Cloudera Runtime 7.1.7 Service Pack 3 to 7.1.9 Service Pack 1, for a GRANT statement involving multiple columns in a table, Impala service used to create one Ranger policy for each column. This issue is now resolved.

CDPD-63596: There are a few compile-time dependencies on spotbugs-annotation, which is LGPL and not allowed by Apache Software Foundation under Category X.

The spotbugs-annotation, an LGPL thirdparty dependency from the Ozone package is now removed.

CDPD-62620: OM checkpoint request pauses all the background workers and then creates a checkpoint after it acquires all the necessary locks. There is a deadlock between OMDBCheckpointServlet#getCheckpoint request and RocksDBCheckpoint#pruneSstFiles background process causing a halt in the OM and no new read/write requests are taken after some time.

This issue is now resolved. The double locking is removed and now there is no deadlock between OMDBCheckpointServlet#getCheckpoint request and RocksDBCheckpoint#pruneSstFiles background process.