Integration, orchestration & processing
dbt
Data transformations in the warehouse via SQL with tests and documentation.
What is it
dbt (data build tool) is a tool for transforming data already inside the warehouse using the ELT approach. You describe data marts as ordinary SELECT queries, and dbt builds the dependency graph between models, materializes them, runs quality tests and generates documentation with lineage. It brings engineering practices — versioning, tests, CI — to SQL models.
Where and why it is used
I use dbt for the transformation layer in a DWH: turning raw data into clean data marts with tests and clear documentation. This makes the logic of metric calculations transparent and reproducible, and changes safe. It pairs well with orchestration on Airflow.
Used in my services
Alternatives
SQL scripts by handDataformSQLMeshStored procedures
Pros and cons
Pros
- Transparent and version-controlled data mart logic
- Built-in quality tests and documentation with lineage
- Familiar SQL — a low barrier to entry for analysts
- Works with BigQuery, ClickHouse, PostgreSQL and others
Cons
- – Transformations only (T), does not handle loading (E/L)
- – Requires discipline and an established process
- – Can fall short for complex SQL-model logic
- – Requires an orchestrator to run on a schedule
