Using Airflow

CDE packages the open source version of Airflow. Airflow is maintained and upgraded at each CDE version update. For example, the version of CDE 1.16 includes Airflow 2.2.5.

CDE Airflow imposes no limitations on Operators, Plugins or other integrations with external platforms. Users are free to deploy Airflow DAGs in CDE as dictated by the use case. Cloudera contributed two operators to the Airflow Community:

CDE Operator: used to orchestrate Spark CDE jobs. This has no requirements other than creating a Spark CDE job separately and then referencing it within the Airflow DAG syntax.
CDW Operator: used to orchestrate CDW Hive or Impala queries. This requires a Cloudera Virtual Warehouse and setting up of an Airflow connection to it.

For an example DAG in CDE using the two operators, see CDE Airflow DAG documentation.

Accessing the CDE Airflow UI

From your Virtual Cluster Service Details page, click Airflow UI to access the UI.

The Airflow DAGs page is displayed.

Using the CDE CLI

For Spark CDE jobs, the CDE CLI provides an intuitive solution to create and manage Airflow CDE jobs.

The CDE CLI commands to create a resource and upload files to it are identical as in the Spark CDE job section.

Submit Airflow CDE job with local DAG file
```
cde airflow submit my-airflow-job.py
```
Create Airflow CDE job
Notice that the --application-file flag has been replaced with the --dag-file flag.
```
cde job create --name myairflowjob --type airflow --dag-file path/to/airflow_dag.py 
```

Using the CDE API

As with the CDE CLI, this section builds on the prior examples on the CDE API with Spark CDE jobs.

The payload now includes the argument airflow for the type parameter. For more details on building Airflow CDE requests, see the Swagger API page.

curl -H "Authorization: Bearer $ACCESS_TOKEN" -X 'POST' \
'$JOBS_API_URL/jobs' \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{"type": "airflow", "airflow": {"dagFile": "my_dag.py"}, "identity": {"disableRoleProxy": true},"mounts": [{"dirPrefix": "/","resourceName": "oozie_migration"}],"name": "oozie2airflow_migration","retentionPolicy": "keep_indefinitely"}'