LogoDuyệtSr. Data Engineer
HomeAboutPhotosInsightsCV

Footer

Logo

Resources

  • Rust Tiếng Việt
  • /archives
  • /series
  • /tags
  • Status

me@duyet.net

  • About
  • LinkedIn
  • Resume
  • Projects

© 2026 duyet.net | Sr. Data Engineer

Airflow control the parallelism and concurrency (draw)

Airflow control the parallelism and concurrency

Airflow configuration to allow for a larger scheduling capacity and frequency:

  • parallelism
  • max_active_tasks_per_dag
  • max_active_runs_per_dag

DAGs have configurations that improve efficiency:

  • max_active_tasks: Overrides max_active_tasks_per_dag.
  • max_active_runs: Overrides max_active_runs_per_dag.

Operators or tasks also have configurations that improves efficiency and scheduling priority:

  • max_active_tis_per_dag: This parameter controls the number of concurrent running task instances across dag_runs per task.
  • pool: See Pools.
  • priority_weight: See Priority Weights.
  • queue: See Queues for CeleryExecutor deployments only.

Credits

  • Airflow Fundamental Concepts > Backfill
  • How to control the parallelism or concurrency of an Airflow installation?
Jul 16, 2023·3 years ago
|Data|
DataData EngineeringApache Airflow
|Edit|
On this page
  • Credits
On this page
  • Credits