Creating Airflow jobs using Cloudera Data Engineering

Learn about how to create Airflow jobs using Cloudera Data Engineering.

An Airflow job in Cloudera Data Engineering consists of an Airflow DAG file and various optional resources. Jobs can be run on demand or scheduled.

important

The default Airflow worker pod resource requests limits are as follows:

CPU requests = 1
CPU limits = No limit
Memory requests = 2 Gi
Memory limits = 2 Gi

Also, you can set the custom resource requests for an Airflow task through executor_config.

For example, sample DAG file is as follows:

from airflow import DAG
from airflow.operators.python import PythonOperator
from pendulum import datetime
from kubernetes.client import models as k8s

config_resource_requirements = {
    "pod_override": k8s.V1Pod(
        spec=k8s.V1PodSpec(
            containers=[
                k8s.V1Container(
                    name="base",
                    resources=k8s.V1ResourceRequirements(
                        requests={"cpu": 0.5, "memory": "1024Mi"},
                        limits={"cpu": 0.5, "memory": "1024Mi"}
                    )
                )
            ]
        )
    )
}


with DAG(
        dag_id="python_operator_custom_resources",
        start_date=datetime(2024, 1, 1),
        schedule=None,
        catchup=False,
):
    PythonOperator(
        task_id="hello_task",
        python_callable=lambda: print("Hello!"),
        executor_config=config_resource_requirements
    )

In the Cloudera Management Console, click the Data Engineering tile and click Overview.
In the Cloudera Data Engineering Services column, select the service that contains the virtual cluster that you want to create a job for.
In the Virtual Clusters column, locate the virtual cluster that you want to use and click the View Jobs icon.
In the left navigation menu, click Jobs.
Click the Create Job button.
Provide the Job Details:
1. Select Airflow for the job type. The available fields on the user interface updates automatically.
2. Specify the Name.
3. Click the File option and select the way you want to provide the DAG file. Using the File option, you can do the following:
  - Upload the DAG file to a new resource.
  - Use the DAG file from a previously created resource.
  - Upload any other resources required for the job.
  For more information about the Editor, see Creating an Airflow DAG using the Pipeline UI.
If you do not want to run the job immediately, click the Create and Run drop-down menu and then click Create. Otherwise, click Create and Run to run the job immediately.

note
If you select the Create and Run option from the Create and Run drop-down menu, the DAG file must not be paused while creating and provide a False value to the is_paused_upon_creation field.