March 27, 2025

This release (1.23.1-H1) of the Cloudera Data Engineering service on Cloudera on cloud introduces the following changes.

This release does not contain new features, but includes the following fixes:

DEX-15814: AirflowAPI list_runs_by_id fails with 'NoneType' object has no attribute 'strftime'🔗

Cloudera Data Engineering Airflow Run Status updates did not work, which could lead to duration mismatch of Job duration versus the actual Airflow Dag Run duration, or it could lead to run status mismatch, or both. This happened when the same Airflow Job was triggered (AdHoc) multiple times and there was a limit on how many parallel DAGs could run at the same time. When the limit was reached, the DAG Runs were queued. As long as there were queued Airflow DAG Runs with None start_date, the Cloudera Data Engineering Job Run status updates did not work. Depending on the workload, this could take a long time. When this issue was present, Cloudera Data Engineering Jobs could not be killed. The state changed to Killed, but it changed back to the previous state. When this issue was present, the airflow-api logs listed: Could not search DAG runs: 'NoneType' object has no attribute 'strftime'.

DEX-15813: Cloudera Data Engineering killed job state can go back to previous state🔗

Jobs could not be killed. When you tried to kill a Cloudera Data Engineering Airflow Job that was in running state, the Cloudera Data Engineering Job state changed to killed for a couple of seconds, but then it changed back to the previous state.

DEX-16304: Eliminate "Failed to parse DAG run start date" error logs when startDate is empty🔗

If the start_date was zero for an Airflow DAG Run, an unnecessary error log was created.

DEX-16232: Fix already existing DAG check in airflow-api🔗

A Cloudera Data Engineering Airflow Job could become corrupted during Cloudera Data Engineering Job creation, making it unusable for further management. The existing Airflow DAG check during Airflow Job creation did not handle OS-related issues properly (for example: NFS mount issue). If it happened, the metadata of an existing DAG could become corrupted. Corrupted Cloudera Data Engineering Airflow Jobs could not be deleted.

DEX-14724: Deletion logic prevents Cloudera Data Engineering Jobs from being deleted🔗

A corrupted Airflow Job could not be deleted through the CLI, the REST API, or the UI.

DEX-15842: Cache the service account for consecutive job launches🔗

When a large number of Jobs were submitted, during the Job launch, the service account check failed due to the Kubernetes API limit and the default timeout. Job submission failed with the following error: could not create user service account: client rate limiter.

DEX-15957: runtime-api pod restart causes RunDagMonitor to fetch all the DAG runs (airflowapi DoS)🔗

Cloudera Data Engineering Airflow Job CRUD operations and Job Run Status updates did not work on Virtual Clusters with big Airflow DAG Run history. The airflow-api crashed after the jobs-api restarted if there were at least 400 000 Airflow DAG Runs in the Virtual Cluster.