Overview
Public vector-DB benchmarks mostly measure unfiltered ANN search on static datasets — conditions that describe approximately zero enterprise RAG deployments. Real workloads have tenant-scoped filters on every query, continuous ingestion, and a p99 SLA. Before recommending a store for a client's 40M-chunk RAG platform, I benchmarked Milvus, pgvector, and Qdrant under those conditions with a fully reproducible harness.
Setup: 10M chunks (1536-dim), 200 simulated tenants with skewed sizes, every query filtered by tenant, and three load profiles (read-only, 90/10 read/write, bulk-reindex-during-reads). Headline finding: the raw-speed ranking inverted under filters and write load. pgvector was slowest at pure ANN but its filtered p99 was the most stable, and its operational story (it's just Postgres) is a feature no benchmark chart captures. Qdrant gave the best filtered-recall/latency balance; Milvus won bulk ingestion and scale-out headroom but cost the most operational attention.
Architecture
- Dockerized harness pins versions and resources (8 vCPU / 32GB per store) with identical HNSW-equivalent parameters where tunable.
- Corpus generator produces 10M chunks across 200 tenants with a Zipf size distribution — tenant skew is where filtered indexes go to die.
- Locust drives three profiles: read-only search, 90/10 read/write, and reads during a bulk reindex; each run is 30 minutes after warm-up.
- Ground truth from exact brute-force search on a 10k query sample; recall@10 computed per run so latency numbers are never quoted without their recall.
- All metrics (p50/p95/p99, recall, ingest rate, RSS memory) scrape into Prometheus/Grafana with one dashboard per store.
- Results tables are generated from raw run data by script — no hand-transcribed numbers anywhere in the report.
Code Snippets
class QdrantAdapter(VectorStoreAdapter):
async def search(self, q: BenchQuery) -> BenchResult:
t0 = time.perf_counter()
hits = await self.client.query_points(
collection_name="bench",
query=q.vector,
query_filter=Filter(must=[FieldCondition(
key="tenant_id", match=MatchValue(value=q.tenant_id))]),
limit=10,
search_params=SearchParams(hnsw_ef=self.ef_search),
)
latency_ms = (time.perf_counter() - t0) * 1000
return BenchResult(
ids=[p.id for p in hits.points],
latency_ms=latency_ms,
recall=recall_at_k(q.ground_truth, [p.id for p in hits.points], k=10),
)-- Global HNSW index for the long tail of small tenants.
CREATE INDEX chunks_embedding_hnsw ON chunks
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 200);
-- The 5 largest tenants (> 500k chunks each) get partial indexes,
-- which kept their filtered p99 from collapsing under skew.
CREATE INDEX chunks_tenant_acme_hnsw ON chunks
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 200)
WHERE tenant_id = 'acme';
SET hnsw.ef_search = 128;
SELECT id, content
FROM chunks
WHERE tenant_id = 'acme'
ORDER BY embedding <=> $1::vector
LIMIT 10;Lessons Learned
- Benchmark your workload, not the vendor's. Every headline ranking inverted once tenant filters and concurrent writes entered the picture.
- Never report latency without recall. Two stores at 'p95 = 40ms' were incomparable until we saw one was running at 0.99 recall and the other at 0.91.
- Filtered search is the great equalizer. HNSW graph traversal under selective filters degrades in store-specific ways that no unfiltered benchmark predicts.
- Operational cost belongs in the results table. pgvector 'lost' on speed but a three-person team already running Postgres ships it months earlier than a new distributed system.
- Tenant skew breaks naive strategies. The Zipf distribution mattered more than corpus size — per-large-tenant partial indexes in pgvector and per-tenant payload indexes in Qdrant were the difference between stable and pathological p99s.