Creating a custom Airflow Python environment (Technical Preview)
To manage job dependencies, Cloudera Data Engineering (CDE) supports creating custom Python environments dedicated to Airflow using the airflow-python-env resource type. With this option, you can install custom libraries for running your Directed Acyclic Graphs (DAGs). The supported version is Python 3.8.
A resource is a named collection of files or other resources referenced
by a job. The airflow-python-env resource type allows you to specify a
requirements.txt file that defines an environment that you can then
activate globally for airflow deployments in a virtual cluster. You can specify any Python
package which is compatible with the Airflow python constraints. These constraints can be
found at
https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt.
The Airflow and Python versions depend on your CDE version.
