Overview
Every prompt tweak on our extraction pipeline was being validated by eyeballing three examples in a playground. Regressions shipped constantly: a wording change that fixed one customer's invoices broke date parsing for another. The team needed the equivalent of a test suite for prompts — deterministic where possible, judged where necessary.
The harness treats prompts as versioned artifacts with attached golden datasets. Deterministic assertions (JSON validity, required fields, enum membership) run first and are free. Semantic quality — is this summary faithful, is this tone right — goes to an LLM judge scoring against an explicit rubric, one criterion per call, with the judge forced to quote evidence before scoring. Results land in DuckDB so we can diff any two prompt versions across the full dataset in CI.
Architecture
- Golden datasets are YAML files colocated with each prompt: input, expected output (for deterministic checks), and rubric criteria (for judged checks).
- Stage 1 runs cheap deterministic assertions — schema validity, field presence, regex checks — and fails fast before any judge tokens are spent.
- Stage 2 runs an LLM judge per rubric criterion (faithfulness, completeness, tone) with a 1-5 scale, forced to cite verbatim evidence before emitting a score.
- Judge outputs are Pydantic-validated; an unparseable judgment is a harness error, never a silent pass.
- Every run writes scores to DuckDB keyed by (prompt_version, case_id, criterion); CI comments a score diff table on the PR.
- A small human-labeled calibration set measures judge/human agreement (Cohen's kappa) so we know when the judge itself has drifted.
Code Snippets
class Judgment(BaseModel):
evidence: str = Field(description="Verbatim quote from the output")
reasoning: str
score: int = Field(ge=1, le=5)
JUDGE_PROMPT = """You are grading one criterion only: {criterion}.
Rubric: {rubric}
First quote the exact evidence from the output, then reason,
then score 1-5. Do not grade anything outside this criterion.
<input>{case_input}</input>
<output>{model_output}</output>"""
def judge_criterion(case: EvalCase, output: str, criterion: Rubric) -> Judgment:
resp = client.messages.create(
model="claude-sonnet-4-5",
max_tokens=500,
messages=[{"role": "user", "content": JUDGE_PROMPT.format(
criterion=criterion.name, rubric=criterion.rubric,
case_input=case.input, model_output=output)}],
)
return Judgment.model_validate_json(extract_json(resp))SELECT
criterion,
AVG(CASE WHEN prompt_version = 'v14' THEN score END) AS v14,
AVG(CASE WHEN prompt_version = 'v15' THEN score END) AS v15,
COUNT(*) FILTER (
WHERE prompt_version = 'v15' AND score < 3
) AS v15_failures
FROM eval_runs
WHERE prompt_version IN ('v14', 'v15')
GROUP BY criterion
ORDER BY (v15 - v14) ASC;Lessons Learned
- One criterion per judge call. A single 'rate this 1-10 overall' judge compressed every failure mode into a meaningless average; per-criterion scores made regressions legible.
- Forcing the judge to quote evidence before scoring cut judge hallucination dramatically — it can't invent a failure it can't quote.
- Deterministic checks first, always. Roughly 60% of regressions were caught by free schema/regex assertions before a single judge token was spent.
- The golden dataset is the real asset, not the harness. Twenty minutes labeling hard cases beat hours of harness engineering every time.
- Calibrate the judge against humans periodically. Ours agreed with human graders at kappa 0.74 initially, then drifted after a model upgrade — without the calibration set we'd never have noticed.