Cloudera Docs

Known issues in 1.5.5 SP4

You might run into some known issues while using Cloudera Data Engineering on premises.

Existing known issues from Cloudera Data Engineering on premises 1.5.5 SP3 and lower releases are carried into Cloudera Data Engineering on premises 1.5.5 SP4. For more details, see

For known issues fixed since Cloudera Data Engineering on premises 1.5.5 SP3, see Fixed issues in 1.5.5 SP4.

DEX-19060: The Configuration tab on Jobs runs does not show the details of data connectors
The Configuration tab on the Jobs runs page does not show the details of data connectors even though the job that was run used a data connector.
You can check the Configuration tab on the Job page, where the data connectors details are visible.
DEX-22871: Restored PySpark job fails with ModuleNotFoundError because python-env is not applied at runtime
After you perform a backup and restore of a Cloudera Data Engineering service, PySpark jobs linked to a python-env resource fail at runtime. Although the restored Python environment resource rebuilds successfully and reaches a Ready status, the system does not correctly apply or mount it onto the driver. As a result, the job cannot import packages present in the restored environment, which causes a ModuleNotFoundError.
None
DEX-23003: Spark History UI may be unavailable after upgrade for Spark jobs killed by the Cloudera Data Engineering upgrade flow
The Spark History UI can be unavailable after an upgrade for Spark jobs killed during the Cloudera Data Engineering upgrade process. After upgrade, spark.sparkAppID may be absent and spark.sparkAppURL may contain .../history/unknown/jobs/, so the Spark History UI does not open.
None
DEX-23208: Spark jobs fail to interact with Ozone after Cloudera Base on Premises upgrade due to stale ozone-parcel-image
When upgrading a Cloudera Base on premises cluster (example, from 7.1.9 to 7.3.2 SP1) while running Cloudera Data Engineering, the upgrade orchestration fails to automatically update the Cloudera Data Engineering ozone-parcel-image. As a result, the environment continues to use a stale image that contains a classpath incompatibility (Protobuf 2.x vs 3.x). This causes Spark jobs or sessions attempting to read from or write to Ozone to fail immediately with a java.lang.NoClassDefFoundError: com/google/protobuf/LiteralByteString.
Upgrade Cloudera Data Services on premises and Cloudera Data Engineering together with the Cloudera Base on premises so image selection runs and ozone-parcel-image matches the new Cloudera Base on premises.
DEX-23420: Airflow Python environments are not backed up during side-by-side upgrades
During a side-by-side upgrade using dex-upgrade-utils (backup and restore), Airflow Python environments (pyenv) are not included in the backup. Consequently, during the restore operation on the target virtual cluster, any Airflow DAG that references packages from the pyenv through top-level imports fails DAG parsing with a ModuleNotFoundError. This causes the job to be rejected and can cause the restore process to hang.
  1. If the restore process hangs due to failed DAG imports, apply the workaround in the following DEX-23421 (Restore operation hangs indefinitely due to missing timeout during partial resource or job restore) issue to allow the restore process to proceed and complete.
  2. After the restore completes, manually recreate and activate the required Airflow Python environment (pyenv) on the target virtual cluster.
  3. Manually recreate the Airflow jobs that fail during the restore process.
DEX-23421: Restore operation hangs indefinitely due to missing timeout during partial resource or job restore
During Cloudera Data Engineering restore operations using dex-upgrade-utils (such as upgrading from Cloudera Data Services on premises 1.5.5 SP2 to SP3 or SP4), if any job or resource fails to restore on the target virtual cluster, the restore operation hangs indefinitely. This occurs because there is no enforced timeout mechanism, and the workflow fails to validate the job or resource count when a partial failure occurs.
Manually create the missing dummy job or resource on the target virtual cluster to satisfy the resource count validation. This allows the restore operation to proceed and complete.
DEX-23695: Spark 3.5 jobs fail with ModuleNotFoundError due to Python version mismatch in builder image
In Cloudera Data Engineering 1.5.5 SP3 CHF3, the Python builder image for Spark 3.5 in Cloudera Base on premises 7.1.9 contains the incorrect Python version (Python 3.10 instead of the expected Python 3.11). As a result, Spark jobs using a python-env resource for Spark 3.5 fail immediately with a ModuleNotFoundError during application import, even when the resource build reports as successful.
Upgrade to Cloudera Data Services on premises 1.5.5 SP4 or later, where the Python builder image uses Python 3.11 matching the Spark 3.5 runtime and rebuild the pyenv.
DEX-23853: Cloudera Data Engineering virtual cluster restore from in-place backup leaves scheduled Spark jobs paused
When restoring a Cloudera Data Engineering virtual cluster from an in-place upgrade backup archive, scheduled Spark jobs remain in a paused state and do not execute automatically. This occurs because the restore process recreates the jobs using the paused state captured during the backup phase, but it lacks a step to resume the schedules after the restore operation completes.
Manually resume or unpause the affected job schedules on the restored virtual cluster after the restore operation completes.
DEX-24011: Clicking the Spark UI tab in Job Run details incorrectly opens the Sharing tab
In the Cloudera Data Engineering UI, when you click the Spark UI tab on the Job Run details page, the frontend incorrectly renders the Sharing tab. Although the browser URL updates to the .../sparkUI path, the UI highlights the Sharing tab and displays the Users and Groups access list. This prevents users from accessing the embedded Spark UI directly from the job run details page.
Click the Logs tab first, then click the Spark UI tab. If the Spark UI does not load on the first attempt, repeat the sequence once more.
DEX-24013: Observability CRN displays pre-upgrade Service and Virtual Cluster IDs for new job runs
When you perform a side-by-side (backup and restore) upgrade of a Cloudera Data Engineering Service and Virtual Cluster, newly executed jobs display an incorrect Observability CRN in the Cloudera Data Engineering Job Runs UI. The CRN retains the pre-upgrade Service ID and Virtual Cluster (VC) ID instead of the new infrastructure IDs. As a result, searching for job runs in the Cloudera Observability UI using the displayed CRN returns no records.
To locate records in the Cloudera Observability UI, replace dex-base-[***UNIQUE-ID***] for the service and dex-app-[***UNIQUE-ID***] for the Virtual Cluster in the displayed CRN with the updated IDs from the Cloudera Data Engineering UI, and then perform the search.
DEX-24014: Job timeline, driver logs, and Spark UI fail to load after same-cluster DRS restore following a side-by-side upgrade
Following a side-by-side upgrade using backup and restore scripts, performing a Data Recovery Service (DRS) backup and restore on the same cluster causes Cloudera Data Engineering UI observability components specifically the job runs timeline, driver logs, and Spark UI to fail to load or display as blank for restored workloads.
No workaround is available once this issue occurs after a DRS restore. To prevent this issue, use an in-place upgrade instead of a side-by-side upgrade with backup and restore scripts.
DEX-24195: Inverted Privacy Settings toggle button logic causes incorrect job sharing permissions
The Access Control List (ACL) logic for the Privacy Settings toggle button in the Cloudera Data Engineering UI is inverted. When you enable Privacy Settings to restrict job access, the system incorrectly applies a global Full Access policy to all users. Conversely, when you disable Privacy Settings to share a job, the system incorrectly restricts access solely to the job creator.
Depending on your configuration:
  • If Privacy Settings are enabled, after creating the job, do the following steps:
    1. In the Cloudera console, click the Data Engineering tile. The Cloudera Data Engineering Home page displays.
    2. In the left navigation, click Jobs.
    3. Click the job for which you want to update privacy settings.
    4. Click the Sharing tab and manually remove the default permission that grants access to all users.
  • If Privacy Settings are disabled, manually add the required sharing permissions using the Cloudera Data Engineering UI, CDE CLI or API to grant access to the intended users.
DEX-24197: Spark 2.4.8 jobs fail with "failed to build run workspace" error after Cloudera Data Engineering cluster restoration or upgrade
Spark 2.4.8 jobs configured using the CDE CLI or API fail at startup with the following error:
failed to build run workspace: could not resolve entry: lstat /usr/include/python3.6m: no such file or directory

This issue occurs when a Python environment (python-env) resource is incorrectly added under general resource mounts (mounts).

Remove the Python environment resource from general resource mounts (mounts) and attach it exclusively using the designated Python environment field:
  • CLI: Use the --python-env-resource-name flag.
  • API: Set the python.pythonEnvResourceName field.
  • UI: Select the relevant Python environment using the UI and verify that it is not listed under Mounts.
DEX-24226: Cloudera Data Engineering service returns 404 error after restoring backup with original service ID
A Cloudera Data Engineering service or Virtual Cluster endpoint returns a 404 Not Found (NGINX) error after restoring a backup with the original service ID. This issue applies to Cloudera Data Engineering on Cloudera Data Services on premises where TLS certificates are issued by certificate manager through a Venafi TPP ClusterIssuer.
No immediate action is required. The certificate manager retries the certificate enrollment automatically with a newly generated key. The first retry occurs 1 hour after the failure, and each subsequent failure doubles the retry interval. If the service does not recover, contact Cloudera Support.
DEX-24444: Cloudera Data Engineering in-place upgrade preflight check fails when retrying a failed EnterMaintenance phase without cancelling the upgrade
During a Cloudera Data Engineering in-place upgrade, if the EnterMaintenance phase fails after virtual clusters enter the AppPaused state, retrying the phase without first cancelling the upgrade causes a preflight validation failure. This occurs because the validation check detects that the virtual clusters are already in the AppPaused state.
Click Cancel Upgrade to cancel the upgrade process, and then restart the upgrade from the maintenance step.