Fixed Issues in Apache Impala

Review the list of Impala issues that are resolved in Cloudera Runtime 7.1.9.

OPSAPS-64734: Unable to add Impala as a new service if the default hive warehouse is on Ozone

Impala creation no longer fails when warehouse directory is on Ozone.

OPSAPS-44763: CM should mark an Impala Query if the query touches Erasure Coding file in HDFS/Ozone

Highlights erasure-coded bytes read in queries and allows filtering on this counter.

OPSAPS-65783: Impala status is red in CDP 7.1.8 with python 3.8

* What was the problem? User would see "Impala Query Monitoring Status Check" alerts and TypeErrors like below in the Cloudera Manager Agent logs [04/May/2023 14:08:06 -0500] 2520 ImpalaDaemonQueryMonitoring throttling_logger ERROR (98 skipped) Error fetching executing query ids at 'http://<hostname:port>/inflight_query_ids' Traceback (most recent call last): File "/opt/cloudera/cm-agent/lib/python3.8/site-packages/cmf/monitor/impalad/query_monitor.py", line 506, in get_executing_query_ids query_ids = _parse_executing_query_ids(opened_url) File "/opt/cloudera/cm-agent/lib/python3.8/site-packages/cmf/monitor/impalad/query_monitor.py", line 331, in _parse_executing_query_ids return query_ids_string.split('\n') TypeError: a bytes-like object is required, not 'str' CM Agent logs would show TypeErrors when user views the Impala query monitoring which appears to be the actual cause for the "Impala Query Monitoring Status Check" alerts in the CM. * What was the fix to the problem? Incompatible Python3 issues fixed for Impala query monitor in Cloudera Manager Agent. * Under what conditions would a user see the problem? If Impala queries are executed and viewed in Cloudera Manager * Do they need to do anything differently now? No

CDPD-26802: Improved self events detection in catalogd cache

This parent jira tracks the issues that fixed the consistency issues in detecting self-events in catalogD's cache by using a flag called 'lastSyncEventId' on the metadata object.

CDPD-47030: Impala-shell ldap_password_cmd fails on Python 3.8

Fixes impala-shell --ldap_password_cmd with Python 3.

CDPD-48721: Impala - Upgrade Jquery Datatables to the latest version to avoid Security issues

Updates Jquery Datatables in the Impala UI to address CVE-2020-28458 and CVE-2021-23445.

CDPD-48780: impala-shell now requires setuptools be manually added

Fixes a regression in CHF3 where impala-shell under Python 2 required installing setuptools.

CDPD-49015: IMPALA-11859 Metric tracking encrypted bytes read

Adds BytesReadEncrypted to query profiles and the metric impala-server.io-mgr.encrypted-bytes-read to observe reads of encrypted data from HDFS/Ozone.

CDPD-50426: Backport IMPALA-11845 to 7.1.9: Fix incorrect check of struct STAR path in resolvePathWithMasking

Fix IllegalStateException when Ranger column-masking/row-filtering policies are applied on a view and the view alias is used together with STAR in the query, i.e. "v.*" when "v" is the alias such a view. See more in IMPALA-11845.

CDPD-49648: Upgrade chart.js to 2.9.4+ due to CVE-2020-7746

Upgrades chart.js in Impala UI to address CVE-2020-7746.

CDPD-50186: IMPALA-11966 Enable cache_ozone_file_handles by default

Enables cache_ozone_file_handles by default to improve scan performance with Ozone.

CDPD-49781: backport IMPALA-11274 to 7.1.9

Limits conjunctive normal form (CNF) rewrite of expressions to cases with simple predicates to fix performance regressions.

CDPD-45661: Support erasure-coding in impala

Reading erasure-coded files from Ozone is now supported with Impala.

CDPD-47206: IMPALA-11730 Add support for spilling to Ozone

Impala can now be configured to spill to Ozone, for example with scratch_dirs="/tmp/scratch,ofs://ozone-scm:9862/tmp".

CDPD-8130: Add HTTP Strict Transport Security (HSTS) for Impala

Adds HTTP Strict Transport Security (HSTS) to Impala UI responses when HTTPS is enabled.

CDPD-47640: Impala erasure coding support

Impala now supports interacting with erasure-coded files in HDFS.

CDPD-47643: Impala SHOW statement to display EC files and policies

Impala's SHOW FILES, SHOW PARTITIONS, SHOW TABLE STATS, and DESCRIBE EXTENDED now display the erasure code policy for files/tables in filesystems that support erasure coding.

CDPD-58002: setup-ranger() failed on 7.1.9.x due to RANGER-2895

This patch does not include any functionality change. It's only related to Impala's own end-to-end test infrastructure.

CDPD-58314: Impala: run test suite with JDK 17

Test ran successfully in https://playground-01.jenkins.cloudera.com/job/impala-private-parameterized/175/. Automated test runs don't need to hold up the release.

CDPD-43746: Support for Ozone erasure coded data

Impala now supports interacting with erasure-coded files in Ozone.

CDPD-55460: Impala - Upgrade Spring Framework to 5.3.27/6.0.8 due to CVE-2023-20863

Spring Framework has been upgraded to 5.3.27.

CDPD-54930: Impala - Support for JDK17 in all sub-components

Impala supports Java 17.

CDPD-50912: Unable to connect to impala-shell if there is a file with special character in user's home directory

impala-shell no longer fails if user's home directory contains special characters.

CDPD-51180: IMPALA-10186 Write invalid parquet PageLocations which table sort by some columns

Impala no longer creates empty parquet pages.

CDPD-41064: IMPALA-11360 Support Java11 in Impala

Impala supports Java 11.

CDPD-56557: Rolling Upgrade 7.1.8 to 7.1.9 tests fail when impala-shell gets error 'TException: Invalid response from catalogd for request TGetPartialCatalogObjectRequest'

This issue was introduced by IMPALA-11350, which added a new field in the middle of Thrift structure TPartialTableInfo and caused backward incompatibility issue between old version of coordinator and new version of catalog server, and new version of coordinator and old version of catalog server. The issue was fixed by checking if the catalog service versions are compatible when registering coordinator and catalog server to a cluster. Incompatible coordinators and catalog server are partitioned in different clusters.

CDPD-50812: IMPALA-11997 impala-shell: base64.encodestring has been removed in python3.9

Impala now works with Python 3.9.

CDPD-56871: Backport fix for IMPALA-12114 to impacted releases

An issue where idle Impala clients using TLS were needlessly disconnected has been fixed.

CDPD-57735: IMPALA-12217 cgroup memory limit detection doesn't work for cgroups v2

Impala now works with cgroups v2.