Apache Airflow
The original open source workflow orchestrator. Python, DAGs, used everywhere.
What it is
Apache Airflow is the original open source workflow orchestrator. Define workflows as Python DAGs (Directed Acyclic Graphs), schedule them, monitor them, and scale them. 1000+ providers, rich web UI, and battle-tested at scale.
Why we recommend it
If you need to orchestrate data pipelines, ETL, ML workflows, or any scheduled jobs, Airflow is the most widely-adopted open source option. Every data tool integrates with it.
When to use it
- Data pipelines (ETL/ELT)
- ML training and batch inference
- Scheduled jobs that have dependencies
- When you need a battle-tested orchestrator
Quickstart
Run this in your terminal. Self-hosts locally with Docker or your preferred setup.
$ pip install apache-airflowSetup
Tags
Integrates with
Common use cases
- →etl
- →data-pipelines
- →ml-workflows
- →scheduled-jobs
Pricing
Free. Apache-2.0 licensed. Managed via Astronomer, AWS MWAA, or Google Cloud Composer.
Related projects
Activepieces
by Activepieces
Activepieces is an open source no-code automation platform. 300+ integrations, visual flow builder, TypeScript pieces, self-hostable, the modern open source Zapier/Make alternative.
Apache Hop
by Apache Foundation
Apache Hop is an open source data orchestration tool. Visual pipeline designer, multi-engine (Spark, Flink, Google Dataflow), metadata injection, Hop GUI for developers. The Pentaho successor.
Argo Workflows
by Argo (Intuit)
Argo Workflows is a Kubernetes-native workflow engine. Define workflows as Kubernetes CRDs, each step is a container. CNCF project, used by Intuit, Adobe, NVIDIA, and every K8s data team.