Review the list of Impala issues that are resolved in Cloudera Runtime
7.3.1, its service packs and cumulative hotfixes.
Cloudera Runtime 7.3.1.500 SP3:
CDPD-87152: System error when renaming tables during metadata
invalidations
7.3.1.500
A system error could occur when you renamed tables at the same
time as metadata invalidation commands. This was caused by the system unexpectedly
encountering a null table object. The issue is now fixed.
Queries sometimes timed out unexpectedly because the set
execution time limit (EXEC_TIME_LIMIT_S) was inaccurately counting time spent in
planning and scheduling.
This issue is resolved by ensuring the query execution time
limit (EXEC_TIME_LIMIT_S) now starts only when the query begins actual execution.
CDPD-80939: Missing equivalence conjunct in aggregation node
with inline views
7.3.1.500
In queries that include filters on the result of a
UNION operation, the planner sometimes removed required conjuncts,
which caused incorrect query results.
CDPD-83207/CDPD-84628: Impala cluster responsiveness during
table renames
7.3.1.500
Performing ALTER TABLE RENAME operations
caused Impala to hold a critical internal lock for too long, which blocked other
DDL/DMLs.
This issue is resolved by ensuring that the critical internal
lock is no longer held during long-running external calls initiated by ALTER
TABLE RENAME operations. This prevents the entire Impala cluster from being
blocked, allowing other queries and catalog operations to proceed without
interruption.
CDPD-81905: Missing tables in Top-N memory usage list on Web
UI
7.3.1.500
Some tables did not appear in the Top-N Tables list on the
/catalog Web UI after metadata loading in On-demand Metadata
mode, such as from a DESCRIBE command. The memory metrics were only
updated when a FULL Thrift object was requested.
The fix includes updating memory usage metrics immediately after
metadata is loaded for an HDFS table, regardless of the Thrift object type
requested.
CDPD-83441: Inconsistent Row Count Output in Impala-shell
7.3.1.500
When running Impala queries, some commands over HiveServer2
protocol, some commands (like REFRESH or
INVALIDATE) did not show the Fetched X row(s) in
Ys output in Impala-shell, even though Beeswax protocol shows them.
This issue was resolved by adding a new option in Impala-shell
called --beeswax_compat_num_rows. When this option is enabled,
Impala-shell now prints Fetched 0 row(s) in along with the
elapsed time for all Impala commands.
When querying metadata from Impala on a cluster with more than a
few nodes, the query might fail with a null pointer exception.
The fix addresses issues noticed while querying metadata from
Impala on a cluster with more than a few nodes. The change forces Iceberg metadata
scanner fragments to be scheduled on the coordinator. A new flag is introduced in the
TPlanFragment thrift struct and if the value is true, then the fragment
is always scheduled on the coordinator.
Addressing IllegalStateException in Complex Views post
upgrade
7.3.1.500
When executing queries that generate runtime filters where the
same column identifier appears repeatedly after upgrading to the 2024.0.18.0-206
runtime, you encountered the following error: IllegalStateException:
null. This issue has been fixed.
Stability Improvement for analytic functions on collections
7.3.1.500
The sorting process incorrectly included unnecessary elements,
causing errors during array operations and leading to frequent query failures. This
issue has been fixed to ensure only complete data entries are used in the sorting
process. This prevents crashes and maintains stable execution of analytic functions on
collections.
CDPD-73187: Impala Ranger audit plugin fails to create audit
logs
7.3.1.500
The fix ensures that the Ranger plug-in in Hive and Impala send
audit events to the Solr service after upgrading Data Lake to a version that requires
SSL for Ranger's audit events.
IMPALA-13170: Database deletion during metadata fetch
7.3.1.500
Running SHOW DATABASES in Impala while
simultaneously dropping a database in Hive caused an
InconsistentMetadataFetchException due to missing database metadata.
Improved exception handling for operations listing databases and
tables. If a database is dropped during metadata fetch, the operation now handles the
missing database.
CDPD-56130: Event Processor failure due to table lock release
error
7.3.1.500
The event processor entered an error state when a table write
lock attempt timed out. Since the lock was not held by the current thread, an attempt to
release it triggered an IllegalMonitorStateException.
The fix added a check to ensure the table holds the write lock
before releasing it, preventing event processor failures and unnecessary table
invalidations.
CDPD-70945: ConcurrentModificationException in
ReloadEvent processing
7.3.1.500
The event processor encountered a
ConcurrentModificationException when handling partition-level RELOAD
events. This occurred because ReloadEvent.isOlderEvent() checked the
partition reload status without holding a table read lock, leading to conflicts with
concurrent DDL/DML operations modifying the partition list.
The issue was addressed by ensuring that the event processor
acquires a table read lock before checking partition reload status, preventing
concurrent modifications from causing errors.
CDPD-72058: Optimized serialization of position delete
records
7.3.1.500
The serialization of position delete records was inefficient,
storing file paths redundantly and increasing memory usage. The process involved
unnecessary copying, leading to larger buffers that required compression before
transmission.
The fix includes optimizing serialization by grouping position
delete records with the same file path.
CDPD-74683: Incorrect skipping of file metadata reload for
ALTER_TABLE events
7.3.1.500
The optimization introduced in IMPALA-12487 skipped file
metadata reload for ALTER_TABLE events if changes in storage
descriptor were trivial. However, some HMS clients modify both table properties and
storage descriptor. If table properties had non-trivial changes (e.g., a location
change), skipping the file metadata reload led to inconsistencies.
The issue was addressed by refining the checks for skipping file
metadata reload in ALTER_TABLE events.
IMPALA-12607: Optimized event fetching by filtering at the
metastore
7.3.1.500
Impala fetched all metastore events and filtered them in its
cache based on DbName/TableName. This approach became a bottleneck when handling a large
number of events.
Impala now directly fetches events specific to the database or
table from the metastore by leveraging the HIVE-27499 change.
CDPD-82303: EXEC_TIME_LIMIT_S incorrectly
includes planning time
7.3.1.500
The EXEC_TIME_LIMIT_S timer was triggered
during the planning and admission (queuing) phases, which could cause queries to fail
before any processing on the backend started.
The issue was addressed by starting the
EXEC_TIME_LIMIT_S only after the query is ready to run on the
backend. This ensures the timer applies only to the actual processing phase.
Queries with a subquery inside an OR condition could return
incorrect results due to an improper join predicate rewrite. The issue was addressed by
skipping the join rewrite when the subquery is part of a disjunctive (OR)
expression.
Invalid cardinality calculation in sortnode's computestats
7.3.1.500
An error occurred during query execution due to an overflow in
the calculation of row limits, causing unexpected failures. The calculation was updated
to prevent overflow and eliminate the error.
CDPD-84214: User authentication consistency for Impala
queries
Impala queries previously failed with an authorization error,
indicating a mismatch between the user authorized on the Knox connection and the session
username. This issue occurred intermittently, typically on the first attempt when
running Iceberg queries, especially when cookie-based authentication was used after an
initial secure authentication setup.
This issue was resolved by ensuring that user identity is
correctly maintained during cookie-based authentication, even after initial secure
setup. This prevents mismatches in user verification, allowing queries to proceed
without authorization errors.
CDPD-68079: Catalog Object Cache Size Calculation in JDK
11+
7.3.1.300
An error occurred in JDK 11+ where the catalog
object cache incorrectly calculated the size of an object, resulting in an exception
message:
java.lang.reflect.InaccessibleObjectException: Unable to make field private jdk.internal.platform.cgroupv1.CgroupV1SubsystemController jdk.internal.platform.cgroupv1.CgroupV1Subsystem.cpu accessible.
The issue has been resolved.
CDPD-80169: File metadata reload incorrectly skipped on ALTER
TABLE
7.3.1.300
Impala skipped reloading file metadata on ALTER
TABLE events if changes in the StorageDescriptor were
trivial, even when table properties had important changes like a new location.
The fix includes updating the logic to ensure file metadata
reload is not skipped when table properties have non-trivial changes.
CDPD-80168: Missing tables in Top-N memory usage list on Web
UI
7.3.1.300
Some tables did not appear in the Top-N Tables list on the
/catalog Web UI after metadata loading, such as from a
DESCRIBE command. The memory metrics were only updated when a FULL
Thrift object was requested.
The fix includes updating memory usage metrics immediately after
metadata is loaded for an HDFS table, regardless of the Thrift object type
requested.
CDPD-79017: Fixed lost exceptions during re-analysis
failures
7.3.1.200
When an AnalysisException occurs during
re-analysis, calling toSql() throws another exception, causing the
original exception to be lost.
The issue was addressed by ensuring the original exception is
not lost when an error occurs during query re-analysis. Now, the error message properly
logs the root cause instead of being replaced by another exception.
CDPD-78366: Session management for Hue logins with Impala
7.3.1.100
When logging into the Hue web UI, users occasionally encountered
the following
error:
The user authorized on the connection 'hue/gateway0.xyz.site@XYZ.SITE' does not match the session username 'hue/gateway1.xyz.site@XYZ.SITE'
This issue arised when a Hue user’s request was routed to a different
Hue backend other than the one that initiated the original HS2 session with Impala.
With Kerberos authentication, Impala restricts session reuse across different hosts or
realms, even if the user remains the same.
To address this, proxy clients like Hue can now reuse HS2
sessions across multiple hosts or realms. This is allowed as these proxy clients can be
trusted. This improvement enhances session management and ensures a smoother user
experience when using Hue.
CDPD-78368: Executor crash during runtime filter generation
7.3.1.100
Enabling the MIN_MAX runtime filter in nested
loop Joins for certain queries caused out-of-bounds access, leading to executor crashes
that prevented any queries from running, disrupting production.
The issue is resolved by ensuring ScalarExprEvaluator properly
calls the Open() function, preventing out-of-bounds access and stabilizing query
execution.
CDPD-78170: Boolean literals in OR conditions are not simplified
as expected
7.3.1.100
Impala currently fails to simplify expressions like "id = 0 OR
false" to "id = 0". The root cause is that the CompoundPredicate generated by
NormalizeExprsRule is not analyzed, preventing SimplifyConditionalsRule from applying
the expected rewrite.
This fix addresses the issue by ensuring that the rewritten
CompoundPredicate is analyzed properly, allowing boolean literals in AND/OR conditions
to be simplified as expected.
CDPD-78177: Conjunct registration issue during query rewrite in
Impala
7.3.1.100
This issue, introduced in version Cloudera Base on premises7.1.9, caused some queries with
expressions that could be rewritten to false to fail with an
IllegalStateException: Illegal reference to non-materialized
slot
The issue was addressed by reverting an optimization that
skipped registering certain conjuncts, ensuring proper registration and assignment
during query analysis.
CDPD-78353: Failed table loads not retried after metastore
recovery
7.3.1.100
Tables failed to load when the metastore was down, and queries
continued to fail even after the metastore was back up, requiring manual
invalidation.
The system now automatically retries loading failed tables when
a query is executed, ensuring successful loads once the metastore is up, without manual
intervention.
CDPD-78175: Incorrect results due to predicate pushdown in join
condition
7.3.1.100
Impala can incorrectly push predicates to scan nodes, causing
wrong results in some join scenarios. For example, a query that should return 0 rows may
return incorrect results when specific predicates are used.
Removing analytic predicates with self-referencing TupleIds
during query execution prevents incorrect pushdown and resolves the issue.
CDPD-78564: Partition value mismatch during INSERT event
processing
7.3.1.100
Impala failed to process some INSERT events due to partition
value mismatches. Hive encodes partition strings in file paths using URL encoding, but
partition strings in HMS events are not encoded. Impala mismatched partitions during HMS
events processing by decoding them incorrectly, causing event processing failures.
The fix includes decoding partition strings only when derived
from file paths and not from HMS events to ensure accurate partition
mapping.
CDPD-78559: Prometheus metric name incompatibility with JDK11
and JDK17
7.3.1.100
Impala metrics derived from JVM MemoryPoolMXBean names contain
apostrophe characters in JDK11 and JDK17. Since Prometheus does not support apostrophes
in metric names, these metrics cannot be consumed by Prometheus.
To address the issue, apostrophes in Prometheus metric names are
now automatically translated to underscores, ensuring compatibility with Prometheus
requirements.
CDPD-78161: Database deletion during metadata fetch
7.3.1.100
Running SHOW DATABASES in Impala while simultaneously dropping a
database in Hive caused an InconsistentMetadataFetchException due to
missing database metadata.
Improved exception handling for operations listing databases and
tables. If a database is dropped during metadata fetch, the operation now handles the
missing database.
CDPD-78363: DDL hangs with SYNC_DDL when Catalogd switches to
standby
7.3.1.100
When Catalogd changes from active to standby while waiting for
SYNC_DDL version, it stops receiving catalog topic updates from the statestore. This
caused DDL queries to hang indefinitely.
To address the issue, Catalogd now regenerates its service ID
when changing to standby and throws an exception if the service ID changes while waiting
for SYNC_DDL version.
CDPD-78365: Queries canceled after statestore failover
7.3.1.100
Queries were canceled after a statestore failover due to
temporary inconsistencies. While a grace period was already in place for statestore
restarts, failovers were not handled the same way.
The fix applies the existing post-recovery grace period to
statestore failovers, preventing query cancellations caused by temporary
disruptions.
CDPD-78563: Configurable TCP keepalive for client
connections
7.3.1.100
Client connections get lost without an explicit close due to
machine resets, network disruptions, or load balancer idle timeouts, leading to resource
leaks in Impala.
To address this, new startup options allow you to enable and
configure TCP keepalive settings for client connections. These options help detect and
close dead connections, preventing premature disconnections.
Incorrect length when inserting multiple CHAR(N) values
7.3.1
When all values in a VALUES clause column are
of type CHAR, but have differing lengths, Impala selects a common
CHAR(max(lengths)) type, padding shorter values with spaces. This
padding can lead to unexpected results if the destination column is
VARCHAR or STRING, as inserting the values
individually would not produce the same padded output.
The VALUES_STMT_AVOID_LOSSY_CHAR_PADDING query option was introduced
to address this discrepancy. When set to true, this option prevents
padding by automatically casting values to VARCHAR with the length of
the longest value in the column, provided all values are CHAR and not
of equal length. By default, this option is set to false.
