November 5, 2026

DEX-24692: Python version mismatch between the dex-runtime-python builder and the Spark-base image for the 7.2.15 stack

Previously, a ModuleNotFoundError error occurred at runtime for PySpark jobs using python-env resources on the Spark 3.2.3 runtime that uses Python 3.9. This issue was caused by a version mismatch: the Resource Builder used Python 3.10 to install packages, while the Spark Driver expected Python 3.9, which led to incompatible site-package paths.

This issue is now resolved. Now the Resource Builder uses a Python version that matches the runtime Python version.

ENGESC-34501: Cloudera Data Engineering upgrade failure caused by exceeding the 1MB Secret limit

Previously, Virtual Cluster creation on AWS failed because the dex-app Helm release Secret exceeded the 1 MB Kubernetes object size limit. This issue occurred because the Amazon RDS global CA bundle was inlined three times.

This issue is now resolved. Region-specific RDS CA bundles with a fallback to the global bundle are now used instead of inlining the global bundle repeatedly. As a result, the dex-app Helm release Secret remains below the 1 MB limit set by Kubernetes.

DEX-22841: Livy pod thread leak in Cloudera Data Engineering 1.26.1 causes job failures

In Cloudera Data Engineering version 1.26.1, Livy pods could experience native thread exhaustion, leading to job failures and HTTP 500 errors. Livy pods could enter a silent complete outage, meaning that all batch and API requests returned HTTP 500, while the pod remained in Running state, preventing Kubernetes from automatically restarting it.

The cause was native thread exhaustion ("unable to create native thread"). A Livy token-refresh bug recreated the Kubernetes client every 5 minutes without closing the previous instance. In Cloudera Data Engineering 1.26, Fabric8 7.x uses Vert.x (~10 native threads per leaked client), causing threads to grow linearly until reaching the cgroup pids.max (~37K) limit. In Cloudera Data Engineering 1.25 and lower versions, the token-refresh leak was present but remained largely latent because Fabric8 used OkHttp.

This issue is now resolved.