Technologies
The tools I use to build analytical data warehouses, pipelines and dashboards. I pick the stack to fit the task, data volumes and budget — here’s what I work with most often.
Data stores & formats
Where data lives: analytical databases, columnar engines and open table formats.
Columnar DBMS for analytics on large volumes with extremely fast queries.
Google's serverless cloud warehouse for petabyte-scale analytics.
Embeddable analytical DBMS — the "SQLite for analytics", runs locally.
Open table format for reliable tables in a data lake and Lakehouse.
Integration, orchestration & processing
How data gets in and becomes marts: ingestion, streaming, orchestration and transformations.
Open EL platform with hundreds of ready-made connectors to data sources.
Data pipeline orchestrator: schedules, dependencies, monitoring.
Data transformations in the warehouse via SQL with tests and documentation.
Distributed platform for real-time event streaming.
A message broker for reliable system integration and task queues.
Data catalog
How to keep data discoverable: catalog, metadata, lineage and quality.
BI & visualization
How data becomes dashboards and decisions: BI platforms and visualization.
