Cloudera Docs

Upgrading Airflow if DAGs and packages are compatible with new Airflow version

You can upgrade Airflow after ensuring the DAGs and the packages in the Airflow Custom Libraries and Operators are compatible with the new Airflow version by using the requirements.txt file.

  1. Get the requirements.txt file for the current Airflow Custom Libraries and Operators.
    After the upgrade, you need the requirements.txt file to restore Python environments. For information on how to get the requirements.txt file, see In-place upgrade with Airflow Operators and Libraries.
  2. Verify the requirements.txt file against an Airflow constraints file.
    For more information, see In-place upgrade with Airflow Operators and Libraries.
    1. If you run into errors during the verification, update the dependencies in the requirements.txt file.
  3. Disable all Airflow Custom Libraries and Operators in all virtual clusters in the CDE service.
  4. Perform the upgrade process.
  5. Using the saved requirements.txt file from step 2, build and activate the Airflow Python environment in the virtual clusters.
    If the build fails because of non-python related dependency issues, you may need to make version adjustments in the requirements.txt file.
  6. On the Airflow UI and CDE jobs UI, verify that the Airflow jobs are running.

    In the case of DAG-related errors:

    • If the DAG parsing fails, look into the Airflow UI to identify the impacted DAGs.
    • If the DAG execution fails, look into the Airflow job logs to identify the impacted DAGs.
  7. Optional: Fix the impacted DAGs or update the dependencies in the requirements.txt file.