Apache Airflow scaling and tuning considerations
There are certain limitations related to the deployment architecture, and guidelines for scaling and tuning of the deployment, that you must consider while creating or running Airflow jobs (DAGs).
CDE deployment architecture
If you use Airflow to schedule and/or develop multi-step pipelines, you must consider the following deployment limitations to scale your environment.
These guidelines describe when to scale horizontally, given total loaded DAGs and/or number of parallel (concurrent) tasks. These apply to all cloud providers.
CDE service guidelines
The number of Airflow jobs that can run at the same time is limited by the number of parallel tasks that are triggered by the associated DAGs. The number of tasks can reach up to 250-300 when running in parallel per CDE service.
Recommendation: Create more CDE services to increase Airflow tasks concurrency beyond the limits noted above.
Virtual Cluster guidelines
- Each Virtual Cluster has a maximum parallel task execution limit of 250. This applies to
both Amazon Web Services (AWS) and Azure.
- For AWS: The recommended maximum number of Airflow jobs loaded per virtual cluster is 1500. As the number of jobs increases, the job page loading time experiences latency along with Airflow job creation time.
- For Azure: The recommended maximum number of Airflow jobs loaded per virtual cluster is 300
to 500. Higher capacity can be achieved but will require manual configuration changes.
Recommendation: Create more Virtual Clusters to load additional Airflow DAGs.
-
The suggested number of Airflow jobs created in parallel is three or less. For jobs that are created in parallel, specify a DAG and/or a task_group for each Airflow operator.
Recommendation: Distribute the creation of Airflow jobs so that you do not create more than three jobs at the same time.
