Fixed issues in 7.1.9 SP1 CHF 1

COMPX-16285: Backport YARN-6523

Optimized the system credentials sent in node heartbeat responses.

CDPD-72911: Backport HDDS-9943

Fixed potential memory leak related to token renewal.

CDPD-72777: It's not possible to search a Hue document with the intention to copy it

The copy action was not provided in the top search or the left assist filter. This issue is now resolved.

CDPD-72775: Copies of shared documents are not visible to the user

Fixed an issue where copies of shared documents were not visible to the users.

CDPD-72731: Backport HIVE-25622

Moved the getURIForAuth method into HiveStorageHandler and changed its signature to accept the Hive Metastore (HMS) table object instead, as it provides implementations with more flexibility around constructing the URIs.

CDPD-72703: Altering a Kudu table with per-range hash partitions might make the table unusable

Fixed an issue where altering a table with per-range hash bucketing by dropping or adding a particular number of columns made the table inaccessible for Kudu client applications.

CDPD-72699: Backport CDPD-70293 to 7.1.9 SP1

Impala executor stopped responding whenever it was restarted. This occurred when extra loads were added to the Impala cluster. This issue is now resolved.

CDPD-72651: Backport CDPD-70019 to 7.1.9 SP1

A shell script is now added to allow users to run authorization-related tests against specified Impala and Ranger patches.

CDPD-72647: Backport IMPALA-13130 to 7.1.9 SP1

Under heavy load, Impala reached maximum memory for data stream operations and DataStreamService did not differentiate between types of requests and rejected requests that helped in reducing the load.This issue is now resolved and Impala now prioritizes data stream operations.

CDPD-72628: SMM UI integration tests are unstable locally

There was flakiness when running the Streams Messaging Manager (SMM) UI integration tests locally. This issue is now resolved.

CDPD-72601: Backport IMPALA-13102 to 7.1.9 SP1

Impala could not load tables with invalid stats due to this,

DROP
                                                  STATS

or DROP TABLE could not be performed on the table. This issue is now resolved.

CDPD-72600: Backport IMPALA-13180 to 7.1.9 SP1

Upgraded the PostgreSQL version to 42.5.5 due to CVE-2024-1597.

CDPD-72595: Backport IMPALA-13107 to 7.1.9 SP1

Imapla executors failed due to invalid TExecPlanFragmentInfo received by executors with instance number equal to 0. This issue is now resolved.

CDPD-72593: Backport IMPALA-13170, IMPALA-9441 to 7.1.9 SP1

An InconsistentMetadataFetchException error occured when running

show
                                                  databases

in Impala while simultaneously executing drop database to drop the newly created database in Hive. This issue is now reoslved.

CDPD-72545: Backport IMPALA-12370 to 7.1.9 SP1 CHF1

An option is now added to customize timezone when working with UNIXTIME_MICROS columns of Kudu tables.

CDPD-72544: Expand security related headers set by SMM

The following security related headers are now added to SMM UI endpoints:

Referrer-Policy
Cross-Origin-Embedder-Policy
Cross-Origin-Opener-Policy
Cross-Origin-Resource-Policy

CDPD-72543: Security headers are not set for static files in SMM

Fixed SMM failure and some security related headers such as, Content-Security-Policy, X-XSS-PROTECTION, X-Content-Type-Options, X-Frame-Options and Strict-Transport-Security are now applied to static files.

CDPD-72500: 7.1.9 SP1 CHF hue failure

Fixed Hue failure on the CDH-7.1.9.1000 branch on CentOS7 and SLES12 platforms.

CDPD-72481: Ranger - Upgrade Graal-SDK to 22.3.5 due to CVE-2024-20918, CVE-2024-20926 and CVE-2024-20932

Upgraded the Graal-SDK version to 22.3.5 due to CVE-2024-20918, CVE-2024-20926 and CVE-2024-20932.

CDPD-72388: [AUTOSYNC] Quota count can go wrong when double buffer flush takes time

Fixed an issue that caused Ozone Quota usage to be incorrectly decremented (below actual quota usage) in a situation where the flush of Ozone Manager's DB was delayed due to heavy load.

CDPD-72230: [AUTOSYNC] Worker queue of FullTableCache cleanup consume much memory

An optimization is now introduced to reduce memory utilization when the Ozone Manager is under excessive load.

CDPD-72207: ll_service_id is empty for an invalid notification type

Fixed the query to fetch the latestInvalidNotificationId even when ll_service_id is empty. This ensures that the NameNode gets the appropriate delta's mappings.

CDPD-72142: [AUTOSYNC] Keys from DeletedTable and DeletedDirTable of AOS should be deleted on batch operation while creating a snapshot

On snapshot creation, the DeletedTable and DeletedDirTable of the Active Object Store (AOS) were cleared. This operation was not done in the same transaction as Snapshot create which caused orphan block objects in case of bootstrapping and lagging follower. This is now resolved and Snapshot creation and clearing of the DeletedTable and DeletedDirTable on AOS are now a single batch operation.

CDPD-72020: [AUTOSYNC] Some containers affected by HDDS-8129 may still be in the DELETING state incorrectly

Some containers reached the DELETING state incorrectly because of HDDS-8129. Due to this, the Storage Container Manager was not able to maintain their redundancy. This issue is now resolved and these containers are now moved back to the CLOSED state so that the Storage Container Manager can handle them correctly.

CDPD-72008: SMM UI - Upgrade Node.js to 22.4.1/20.15.1/18.20.4 due to multiple CVEs

Upgraded the Node.js version in Streams Messaging Manager UI to 20.15.1 due CVE-2024-27980, CVE-2024-22020, CVE-2024-36137, CVE-2024-22018and CVE-2024-37372.

CDPD-71994: Including search within document as we used to have in CDH

In CDP, there was a unified search at the top, and the results did not display in a tabular format. And these resultant workflows could be copied. This issue is now resolved and the search is now similar to CDH, where the search result is displayed in a tabular format snd by selecting the results, the workflows can be copied.

CDPD-71974: [7.1.x] Spark - OpenCSVSerde treats blank value as null

The missing values in OpenCSVSerde are now considered as empty string.

CDPD-71969: [snapshot] Snapshot create requests failing with Ozone Manager failover error in a system with 30000 snapshots

The snapshot directory could not be created inside the double buffer flush thread due to a deadlock between the DoubleBuffer and the StateMachine. Ozone Manager must be restarted to get out of the deadlock.

CDPD-71868: [Upgrade] Recon Fails to start during Ozone Start

Recon failed to start after upgrade from 7.1.7 SP3 to 7.1.9 SP1 due to Ozone Manager DB layout upgrade. This issue is now resolved and Recon fetches a full snapshot from the Ozone Manager.

CDPD-71807, CDPD-69782: [719 SP1 CHF1 CLONE] Users observing role change from ROLE_SYS_ADMIN to ROLE_USER

The updateUserRoleAssignments function in Ranger-admin reset the role of the user from admin to user role for users which were part of the request but were not part of the same page when paged requests were sent to Ranger-admin from Ranger-usersync. This issue is now resolved.

CDPD-71781: [AUTOSYNC] Move SstFiltered flag to another SnapshotProperties table

The SSTFilteringService updates the snapshotInfoTable that caused snapshot chain corruption if, Ozone Manager crashed before DB got flushed. This interfered with SnapshotInfo updated during SnapshotPurgeRequest. This issue is now resolved.

CDPD-71738: Port CDPD-67520 to 7.1.9 SP1 CHF1

The custom Kudu JSON Web Token (JWT) Principal claim is now usable for HBase. The


                                                  hbase.security.oauth.jwt.token.principal.claim

configuration property allows the use of a Subject/Principal claim different from the default sub.

CDPD-71658: Hue QP MySQL Connector/J license issue

The MySQL connector/J jar is not included in the GBN.

CDPD-71433: Handle connect logical type null values in AvroConnectTranslator

When the time.precision.mode : connect property was provided to the Debezium connector, the connect logical types were used and null values were not handled, that caused a Null Point Exception (NPE). This issue is now resolved.

CDPD-71403: Consistent expand icon throughout the UI

All tables in the SMM UI are now adjusted to use the same expand icon.

CDPD-71402: Active or Inactive status does not have high contrast

The producer/consumer status text color in the SMM UI is now adjusted to be more distinguishable.

CDPD-71400: High contrast throughout the UI, fonts are smaller

Adjusted font sizes and colors throughout the whole application of SMM UI to improve usability.

CDPD-71399: Create Topic Modal issues

Fixed visual issues in the topic creation modal.

CDPD-71398: Create connectors should be splitted by tabs

The connector classes in the creation wizard are now split into tabs.

CDPD-71397: Connector metrics are misaligned

Fixed a visual issue where conntector metrics were not properly aligned.

CDPD-71362: [7.1.9 SP1 CHF1] Temporarily disable the tasks tab on Entity Detail page

In the Entity Detail page, the API of the Tasks tab now displays information depending on the server side property atlas.tasks.ui.tab.enabled. This property was set to false previously.

CDPD-71340: There should be a visual clue when hitting restart on Kafka Connect

Added a visual feedback when connectors are restarted in the form of a notification.

CDPD-71339: Classnames on the Kafka Connect popup are not wrapped

Previously, the connector classnames overflowed popups. This visual issue is now resolved.

CDPD-71338: Menu items flow under some tables

The sidebar popups were previously hidden by the main content. This visual issue is now resolved.

CDPD-71336: Filter selector styling

The dropdowns on the main page are now modified to make the counts more prominent.

CDPD-71335: Listing page table styling

Modified the table styling to for better visual.

CDPD-71333: Wrong table headers are sticky after topic/broker dropdown

Listing page table headers are now sticky instead of the nested table headers.

CDPD-71332: The text in the sidepanel column headers are oversized

Fixed a visual issue where text would overflow in the Kafka clients sidebars.

CDPD-71117: Oozie server does not pass action start time to action conf causes a restarting launcher doesn't find child apps

Whenever Yarn restarted the Oozie Launcher AM, Oozie could not find the previously started child jobs due to a missing original start timestamp from the Oozie Server. And the previously started child Jobs were not terminated when the Launcher AM was restarted. This issue is now resolved.

CDPD-71062: The search component does not work with a regexp

Added a checkbox to the listing page searchbar that toggles the regex search.

CDPD-70955: Ozone - Upgrade Axios version to 0.28.1/1.7.2 due to CVE-2023-45857

Upgraded the Axios version to 1.7.2 due to CVE-2023-45857.

CDPD-70838: Impala session is hanging during catalog and statestore HA testing

A query session on Impala did not respond when the catalogd service failed due to a standby instance if CatalogD HA was enabled. This issue is now resolved.

CDPD-70456: Ozone Recon - Solr Health Check API throws error in upgrade from 7.1.7 to 7.1.9 version

During an upgrade from 7.1.7 to 7.1.9 version, Solr service flag was not enabled by default even when the Solr server was running and the Solr service run was successful. Due to this Solr provider impl class failed to initialize and load, and the Solr health check API did not check if the provider was null, leading to a NPE. The Recon Overview UI did not load due to the NPE. This issue is now resolved by adding a null check.

CDPD-70422: Cannot enforce Oozie parameter oozie.http.hostname

A new property named oozie.http.hostname.override is now introduced to specify the interface that the Oozie Server must be using.

CDPD-70409: Recon Overview Page UI fails to load if Recon Solr Health throws error

Fixed an issue where the Recon UI failed to load when there was an error in the Solr Health check API.

CDPD-70357: [7.1.x] Do not call HMS to get list of pruned partitions when translated filter is empty

Minimized the calls to Hive Metastore (HMS) layer to get the partitions list by making one call for each table irrespective of repetition.

CDPD-69849: HPL/SQL: Backport HIVE-28253 to 7.1.9 SP1

As a part of error handling in HPLSQL mentioned in Error Handling in HPL/SQLthere was a syntax error when trying to set a value for hplsql.onerror using the SET command. This error occurred because, as per the current Hplsql.g4 grammar file, SET command took only an identifier in which dot(.) was not allowed. Hence,

SET
              hplsql.onerror

did not work as per the grammar file. This issue is now resolved and the grammar file is changed to use a qualified identifier to accept dot(.).

CDPD-69848: Execute immediate 'select count from tbl' throwing ClassCastException in hplsql mode.

There was a java.lang.ClassCastException error when SELECT count(*) from result was executed. This is ClassCastException is caused because, the select count(*) query returns a long value but HPLSQL expects a string type value.

As a workaround, instead of using EXECUTE IMMEDIATE 'SELECT count(*) from result', use the query directly SELECT count(*) from result.

CDPD-69847: Signalling CONDITION HANDLER is not working in HPLSQL

The CONDITION HANDLERs defined by a user were not invoked when the signal was given to the corresponding condition. As a part of processing the exception/error handling conditions, user-defined conditions are also processed (just logging into the log file). And, when it has to invoke user-defined condition handlers there are no condition handlers in the stack.

CDPD-69846: HPL/SQL: Backport HIVE-28214 to 7.1.9 SP1

Previously, HPLSQL did not use the Hive variables that were passed through Beeline but used the --hivevar option. This issue is now resolved by modifying the code to use the Hive variables passed through Beeline.

CDPD-68849: Set kerberosEnableCanonicalHostnameCheck=false in beeline on CDP Base

Previously, there was no support when using Kerberos authorisation when connecting to the Hive Virtual Warehouse (VW) using Private Cloud Base Beeline. This issue is now resolved and kerberosEnableCanonicalHostnameCheck is now set to false.

CDPD-67602: Hue - Upgrade PostgreSQL to 42.5.5/42.6.1/42.7.2 due to CVE-2024-1597

Upgraded the PostgreSQL version to 42.5.5/42.6.1/42.7.2 due to CVE-2024-1597.

CDPD-66706: HDFS file compression in Hue is failing

Hue HDFS file compression feature did not work and failed with an error message in the OOZIE application logs. This issue is now resolved.

CDPD-66321: IMPALA-12559 Support x5c Parameter in JSON Web Keys (JWK)

Impala coordinator failed to parse in JSON Web Key Set (JWK) and failed to start the coordinator. This issue is now resolved.

CDPD-62360: When Ranger authorisation is disabled for Solr, we get NPE when calling Solr API using user which does not have permission

There was an exception when calling Solr API when Ranger authorisation was disabled for Solr. This issue is now resolved.

CDPD-61728: Upgrade node.js to fix high cve

Upgraded the Node.js version due to CVE-2021-3450, CVE-2021-44531, CVE-2023-32004 and CVE-2023-32006.

CDPD-57994: Hue import not creating External tables

Previously, there was restriction to create only managed tables with Parquet and Optimized Row Columnar (ORC) formats. This issue is now resolved and external tables can now be created.

CDPD-57931: Upgrade OpenJPA in Oozie, because of CVE

Upgraded the OpenJPA version to 3.2.2 due to CVE.

CDPD-49745: Expand app_path column in *_JOBS tables to allow HDFS paths longer than 255 characters

The APP_PATH column now supports storing paths longer than 255 characters.

CDPD-48664: Retry mechanism anomaly in Oozie with High Availability enabled

There was an issue with the retry mechanism in Oozie when High Availability was enabled. This issue is now resolved.

CDPD-48112: Hue - Upgrade Bootstrap to 3.4.1 or 4.3.1+ due to CVE-2019-8331

Upgraded the Bootstrap version 4.3.1 due to CVE-2019-8331.

CDPD-68483: Avro: hadoop-client version conflict

This fix resolves an issue related to the avro-tools uber JAR having two hadoop-client JARs with different versions, which was causing a conflict.