Upgrade considerations if Cloudera Data Engineering services include Apache Airflow workloads

If your Cloudera Data Engineering version and services are eligible for the in-place upgrade, consider these Apache Airflow-related instructions before performing the in-place upgrade.

Airflow jobs included in the Cloudera Data Engineering service
- Set the catchup option of every Airflow job to false before starting the in-place upgrade. If you do not set the catchup option to false, and if the in-place upgrade fails, you might want to do a manual Cloudera Data Engineering service recovery through the backup. After the backup is restored, any Directed Acyclic Graph (DAG) whose catchup is not set to false might replay its entire run history from the defined DAG start date, which is an undesired behavior in most cases.
- Set the catchup option to false for every DAG as described in the DAG runs official Airflow documentation.
Airflow Variables and Connections included in the Cloudera Data Engineering service
If you use Airflow Variables and Connections in the Cloudera Data Engineering service, the default backup taken before the upgrade does not include Airflow Variables and Connections.
- important
  If your in-place upgrade fails and you perform the backup and restore, your Airflow Variables and Connections are lost. As a safety measure, before the upgrade, Cloudera recommends using the backup-restore-based upgrade script to take a backup.
  Alternatively, you can follow the in-place upgrade workflow, and manually back up and restore your Airflow Variables and Connections if the in-place upgrade fails.
- If your in-place upgrade is successful, your Variables and Connections are kept.
Airflow and Airflow-Python versions after in-place upgrade

On Cloudera Data Services on premises, Apache Airflow remains at version 2.11.2 across the upgrade from Cloudera Data Engineering 1.5.5 SP3 to 1.5.5 SP4. Because the Airflow and Airflow-Python versions do not change on this upgrade path, if you are confident in the compatibility of your DAGs and libraries, follow Upgrading Airflow if DAGs and packages are compatible with new Airflow version.