The analytics stack that most mid-sized teams actually need is smaller than the one they buy. Data lands in object storage as parquet, someone needs to query it, and the default answer has been a warehouse with a monthly bill or a cluster with a maintenance burden. An open table format and an embedded engine now cover a large share of that ground without either.
The two pieces do different jobs. Iceberg — or an equivalent open table format — adds a metadata layer over parquet files that provides atomic commits, snapshots, time travel and schema evolution. DuckDB is a fast columnar engine that runs inside your process and reads directly from object storage. Together they are a lakehouse that fits in a container.
What the table format actually fixes
A directory of parquet files is not a table. Two writers produce a partially-visible mess, a reader mid-write sees an inconsistent set, schema changes are a rewrite, and there is no way to ask what the data looked like last Tuesday. The metadata layer fixes all four: a commit is an atomic pointer swap, readers always see a consistent snapshot, columns can be added and renamed without rewriting data, and every snapshot remains addressable until you expire it.
Time travel is the feature that pays for itself in incidents. When a pipeline writes bad data, the recovery is reading the previous snapshot and rewriting from it — not restoring a backup and reconciling everything that happened since.
-- No cluster, no warehouse: an embedded engine reading object storage directly.
INSTALL iceberg; LOAD iceberg;
INSTALL httpfs; LOAD httpfs;
CREATE SECRET lake (TYPE s3, PROVIDER credential_chain, REGION 'eu-west-1');
-- Query the current snapshot of a table that other tools also write to.
SELECT account_id,
date_trunc('day', ts) AS day,
sum(amount) AS revenue
FROM iceberg_scan('s3://lake/warehouse/sales.events')
WHERE ts >= now() - INTERVAL 30 DAY
GROUP BY ALL
ORDER BY revenue DESC
LIMIT 50;
-- Time travel: what did this table say before last night's load?
SELECT count(*) FROM iceberg_scan('s3://lake/warehouse/sales.events',
snapshot_from_timestamp => '2026-09-07 00:00:00');The operational shape of this is what makes it appealing. The query engine is a library, so the same code runs in a scheduled job, a container, a notebook or a test. There is no cluster to keep alive between runs, and the compute cost is a task that exists for ninety seconds.
Where it stops being the right answer
- High-concurrency serving. An embedded engine per process does not replace a warehouse answering hundreds of simultaneous dashboard queries.
- Maintenance is now yours: compaction of small files, snapshot expiry, and manifest upkeep are real jobs that must be scheduled.
- Concurrent writers need a proper catalog to arbitrate commits. Two jobs writing the same table without one will lose data.
- Governance — row-level security, column masking, audit — is thinner than in a managed warehouse, and in regulated settings that gap is the decision.
- Beyond a few terabytes per query, distributed execution wins again. Know where your line is by measuring rather than assuming.
The small-file problem is the lakehouse tax. Skip compaction for six months and your fast query engine will be reading four hundred thousand tiny files.
What I like most about this arrangement is that it is not a commitment. The data sits in open parquet under an open metadata format, readable by every serious engine. If the workload outgrows an embedded engine, you point a distributed one at the same tables and change nothing about how data is stored — which is the opposite of the position you are in when the data lives inside a proprietary warehouse.