Every team building on models arrives at the same wall. The prototype was evaluated by reading outputs and nodding. That does not scale past a few dozen cases, it does not survive a prompt change, and it produces no signal at all once real users are sending traffic you never anticipated.
The infrastructure that replaces nodding has two halves: an offline suite that gates changes, and online monitoring that tells you what is happening to quality right now. Most teams build the first, skip the second, and are consequently blind to the failures that matter — the ones that appear in production usage patterns their golden set does not contain.
Making a judge you can trust
Using a model to score outputs works, but only with the same rigour you would apply to a human annotator. A vague instruction to rate helpfulness from one to ten produces numbers that cluster around seven and move for no reason. A judge with a specific binary question, a written rubric, and required evidence produces scores that correlate with human judgement well enough to act on.
JUDGE = """You are checking one property of an answer. Nothing else.
Property: every factual claim in the answer is supported by the provided sources.
Process:
1. List each factual claim in the answer.
2. For each, quote the supporting source text, or write UNSUPPORTED.
3. Verdict: PASS only if every claim is supported. Otherwise FAIL.
Style, tone, completeness and helpfulness are OUT OF SCOPE. Ignore them."""
class Judgement(BaseModel):
claims: list[ClaimCheck] # evidence BEFORE the verdict, deliberately
verdict: Literal['PASS', 'FAIL']
# Calibrate before you trust it: agreement with human labels on a held-out set.
# Below ~85% agreement the judge is measuring itself, not your system.
assert cohen_kappa(judge_labels, human_labels) > 0.7One property per judge is the rule that makes this work. A single judge asked to weigh faithfulness, tone and completeness into one score produces a number whose movement you cannot attribute to anything. Three narrow judges tell you which thing regressed, which is the only form of the information that leads to a fix.
Online is where the real failures live
Your golden set is a snapshot of what you imagined users would ask. Production traffic drifts away from it immediately: new customer segments, a feature launch that changes phrasing, a document corpus that grew a category. Sample live traffic, run the same judges over it, and track the scores as time series alongside cheap proxy signals — retry rate, conversation abandonment, escalation to a human, thumbs-down.
- Sample stratified by use case, not uniformly; the rare high-value path is the one worth watching.
- Alert on distribution shift in inputs as well as scores. Inputs change before outputs get worse.
- Keep a frozen regression set that never changes, so you can compare across months without moving the ruler.
- Version everything — prompt, model, retrieval config, judge — and attach the versions to every score, or comparisons are meaningless.
- Re-calibrate judges when you change the judge model. A silent judge upgrade shifts every historical baseline.
An eval suite that only runs before deploy measures the questions you thought of. Production measures the ones you did not.
The organisational pattern that works is boring and effective: a weekly review where someone reads twenty sampled production interactions in full. Judges catch regressions on properties you already defined; a human reading real traffic finds the failure mode nobody had named yet, and that is how new judges get written. Automated evaluation is the safety net, not the discovery mechanism.