Data stores & formats
Apache Iceberg
Open table format for reliable tables in a data lake and Lakehouse.
What is it
Apache Iceberg is an open table format for huge analytical tables on top of object storage (S3, GCS, etc.). It adds to data-lake files what they were missing: transactions, schema evolution, time travel and version snapshots. In essence, Iceberg turns a set of files into a reliable table that different engines work with — Spark, Trino, ClickHouse, DuckDB.
Where and why it is used
I use Iceberg when building a Lakehouse, when you need to keep data in an open format and work with it from different engines without duplication. The format removes vendor lock-in and simplifies managing large historical data in the warehouse.
Used in my services
Alternatives
Delta LakeApache HudiClassic DWH without a data lake
Pros and cons
Pros
- ACID transactions and version snapshots on top of a data lake
- Schema and partition evolution without rewriting data
- Open format — works with different engines
- No lock-in to a single vendor
Cons
- – Requires a query engine and a catalog on top of the format
- – Harder to operate than a ready-made DWH
- – Overkill for small data volumes
- – The ecosystem is still actively evolving
