Data stores & formats
DuckDB
Embeddable analytical DBMS — the "SQLite for analytics", runs locally.
What is it
DuckDB is an embeddable columnar analytical DBMS, often called the "SQLite for analytics". It runs inside the application process, without a separate server, and executes complex SQL queries directly over Parquet and CSV files and in-memory tables. It is ideal for fast analytics on a single machine.
Where and why it is used
I use DuckDB for pre-project data analysis and profiling, prototyping data marts and processing extracts without standing up a heavy warehouse. It helps quickly test hypotheses and data quality at the analysis stage, as well as in local ETL scenarios.
Used in my services
Alternatives
ClickHouse (local)SQLitepandas / PolarsApache Spark
Pros and cons
Pros
- Requires no server — runs in a single process
- Reads Parquet and CSV directly, plays well with Python
- Very fast on analytics within a single machine
- Free and easy to install
Cons
- – Not for multi-user concurrent workloads
- – Limited by the resources of a single machine
- – Not a replacement for a centralized DWH
- – Young ecosystem compared to mature DBMSs
