Integration, orchestration & processing
Apache Airflow
Data pipeline orchestrator: schedules, dependencies, monitoring.
What is it
Apache Airflow is a platform for orchestrating data pipelines. Workflows are described in code in Python as a DAG (a graph of tasks with dependencies), and Airflow runs them on a schedule, tracks execution order, retries failed steps and shows status in a web interface. In essence, it is the conductor of scheduled loads and transformations.
Where and why it is used
I use Airflow as the orchestrator in warehouses: scheduling loads from sources, running dbt transformations, data quality checks and dependencies between steps. It is what regular and predictable refreshing of data marts relies on, including in MLOps pipelines.
Used in my services
Alternatives
DagsterPrefectMagedbt Cloudcron + scripts
Pros and cons
Pros
- Flexibility: pipelines are described in code in Python
- A mature industry standard with a large community
- Clear monitoring, retries and alerting
- Many ready-made integrations (operators)
Cons
- – Requires deploying and maintaining infrastructure
- – Overkill for a couple of simple tasks
- – Not designed for real-time stream processing
- – A steeper learning curve than cron
