Every RAG demo looks the same: embed some PDFs, throw cosine similarity at a question, stuff the top five chunks into a prompt. It works on the three questions you rehearsed. Then a real user asks about a clause split across two pages, or uses an internal acronym that never appears in the documents verbatim, and the system confidently answers from the wrong context.
I've shipped RAG into enterprise SaaS where the users are compliance teams, not hackathon judges. The gap between demo and production isn't the model — it's four unglamorous decisions: how you chunk, how you retrieve, how you rerank, and how you measure. Here's what I've learned about each.
Chunking is a data modeling problem, not a splitting problem
Fixed-size 512-token chunks are the default because they're easy, not because they're right. The failure mode is structural: a policy document's exception clause lands in a different chunk than the rule it modifies, and retrieval returns the rule without the exception. That's not a relevance miss — it's a correctness bug.
What works: chunk along document structure (headings, list items, table rows), keep a parent-child hierarchy so you can retrieve small and expand to the enclosing section at generation time, and prepend a breadcrumb of the heading path to every chunk before embedding. That last trick alone fixed a whole class of failures where a chunk said 'the limit is 30 days' and nothing in the chunk said which limit.
Hybrid retrieval, because embeddings can't spell
Dense retrieval is great at paraphrase and terrible at exact tokens. Product codes, error IDs, people's names, internal acronyms — embeddings smear them into a semantic soup. BM25 nails exact matches and fails at paraphrase. You need both, fused. Reciprocal Rank Fusion is embarrassingly simple and consistently beats either retriever alone:
def rrf_fuse(dense_hits, sparse_hits, k=60, top_n=20):
scores: dict[str, float] = {}
for hits in (dense_hits, sparse_hits):
for rank, doc_id in enumerate(hits):
scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank + 1)
ranked = sorted(scores.items(), key=lambda x: x[1], reverse=True)
return [doc_id for doc_id, _ in ranked[:top_n]]
# Retrieve wide from both indexes, fuse, then rerank the fused set.
dense = vector_index.search(query_embedding, limit=50)
sparse = bm25_index.search(query_text, limit=50)
candidates = rrf_fuse(dense, sparse)In one deployment, adding BM25 alongside pgvector lifted recall@20 on our eval set from 71% to 89%. Most of the recovered queries contained identifiers — exactly the queries enterprise users ask most.
Rerank late, retrieve wide
Bi-encoders (your embedding model) score query and document independently, so they can't model interaction between the two. Cross-encoder rerankers read them together and are dramatically better at 'is this chunk actually about the question' — they're just too slow to run over the whole corpus. So don't: retrieve 50 candidates cheaply, rerank to 5-8 with a cross-encoder, and pass only those to the LLM. The latency cost is 100-200ms; the precision gain is the difference between the model answering from the right clause and hallucinating around a plausible-looking neighbor.
- Retrieve 40-60 candidates from hybrid search — recall is cheap at this stage.
- Cross-encode rerank down to a context budget you chose deliberately, not 'whatever fits'.
- Deduplicate near-identical chunks before the prompt; retrieved redundancy crowds out coverage.
- Log retrieved chunk IDs with every generation — you cannot debug RAG without knowing what the model saw.
Evals are the product
You cannot tune what you don't measure, and 'it seems better' does not survive a prompt change. Build a golden set of 50-150 real questions with annotated source passages, then track retrieval metrics (recall@k, MRR) separately from generation metrics (faithfulness, answer relevance via LLM-as-judge with a fixed rubric). Separating the two matters: most 'the AI is wrong' bugs I've triaged were retrieval bugs, and generation-level evals alone would have pointed the fix at the wrong layer.
A RAG system without an eval suite isn't a system — it's a demo you haven't broken yet.
Run the suite in CI on every change to chunking, embedding models, or prompts. When we did this, a 'harmless' chunk-size tweak showed up as a 9-point recall regression before it ever reached a user. That's the whole game: make quality regressions as visible as type errors.