Business Problem
Operations teams manually keyed data from leases, inspection reports, insurance certificates, and compliance documents into the platform, roughly 40k documents a month across customers, at an average of 9 minutes per document. Data entry errors on financial fields like rent amounts and bond values caused downstream accounting corrections, and onboarding a new customer's historical document archive took weeks of temp labor.
Traditional template-based OCR extraction had been tried and failed: document layouts varied wildly across thousands of agencies, property managers, and insurers, and template maintenance became its own full-time job. The business case was straightforward, cut manual keying dramatically, but the hard requirement was trust: wrong data written confidently is worse than no automation, so the system had to know when it was unsure.
Architecture
I designed a multi-stage pipeline orchestrated with LangGraph. Documents entered via API or email ingestion, went through preprocessing (OCR for scans, layout-aware text extraction for digital PDFs), then a classification stage routed each document to a type-specific extraction graph. Extraction used OpenAI models with structured output schemas per document type, followed by a deterministic validation stage checking cross-field consistency, date sanity, and financial arithmetic.
The confidence system was the architectural centerpiece. Every extracted field carried a confidence score derived from model log-probabilities, validation outcomes, and agreement between a fast primary extraction and a cheap verification pass. Documents where all critical fields cleared threshold flowed straight through; anything below routed to a human review UI showing the source region alongside the extracted value for one-click confirm or correct.
Corrections fed a feedback store that served two purposes: few-shot example retrieval via pgvector, similar previously-corrected documents were embedded and injected into prompts for hard cases, and a weekly evaluation harness that scored pipeline versions against a growing golden dataset before any prompt or model change shipped.
System Design
Type-specific extraction graphs behind a common interface, so adding a new document type meant a new schema and prompt set, not pipeline surgery.
Confidence-gated human-in-the-loop: 78% of documents fully automated, 22% routed to review, with the threshold tunable per field criticality.
Deterministic validation layer (cross-field checks, arithmetic, date logic) catching LLM plausible-nonsense errors that no confidence score would flag.
pgvector-backed retrieval of similar corrected examples for dynamic few-shot prompting, lifting accuracy on rare layouts.
Golden-dataset evaluation harness in CI, blocking any prompt or model change that regressed field accuracy.
Queue-based processing with per-customer fairness lanes, so one customer's 10k-document backfill never delayed another's daily flow.
My Responsibilities
Owned the pipeline architecture and led a team of three engineers building it.
Designed the confidence scoring model and tuned automation/review thresholds jointly with operations leadership against an agreed error budget.
Built the LangGraph orchestration, the structured-output extraction stages, and the validation rule engine.
Created the evaluation harness and golden dataset workflow that made prompt changes safe to ship weekly.
Implemented cost controls: model tiering, prompt caching, and batch processing that held unit cost at $0.04 per document.
Instrumented the pipeline with OpenTelemetry traces spanning ingestion to database write, making per-stage latency and failure attribution routine.
Implementation
The first month was deliberately unglamorous: ingestion, OCR normalization, and a rock-solid document store with immutable originals and versioned extraction results. Every later capability leaned on being able to reprocess any document against any pipeline version and diff the outcomes.
Extraction prompts were treated as versioned artifacts with the same review discipline as code. Each document type had a schema, a prompt, and a test suite of golden documents; CI ran the full evaluation on every change and published an accuracy diff to the PR. This is what made weekly iteration safe, and it caught two would-be regressions from model version upgrades.
The review UI mattered as much as the models. Reviewers saw the source page region highlighted next to each low-confidence field, corrected in place, and their throughput hit 45 seconds per document versus 9 minutes for full manual entry. Corrections flowed back automatically into the few-shot retrieval store, so the system measurably improved on exactly the layouts it had been weakest on.
Results
Automated 78% of ~40k monthly documents end to end; overall manual effort down 62% accounting for review time.
Field-level accuracy of 98.1% on critical fields, with undetected error rate under 0.2% verified by ongoing sampled audits.
Customer historical-archive onboarding dropped from 3-4 weeks of temp labor to 2-3 days of pipeline time plus review.
Unit cost held at $0.04 per document through model tiering and section-targeted extraction.
Review throughput of 45 seconds per flagged document, a 12x improvement over full manual keying.
Six new document types added by a single engineer each in under a week, validating the extensibility bet.
Lessons Learned
In extraction systems, calibrated uncertainty is the product; raw accuracy is table stakes.
Deterministic validation catches the LLM errors that statistical confidence never will; always layer both.
An evaluation harness wired into CI is the single highest-leverage investment in any LLM system, it converts prompt engineering from art to engineering.
Shadow mode against the existing human process buys organizational trust that no benchmark can.
Design the correction feedback loop on day one; it is the moat that makes the system better every week.
Future Improvements
Fine-tune a smaller model on the accumulated corrected dataset to cut cost and latency on the five highest-volume document types.
Add table-structure-aware extraction for financial schedules, the current weakest document class.
Active-learning review routing that prioritizes documents whose corrections would most improve the model, not just lowest confidence.
Expose extraction-as-an-API for customers to push documents programmatically with webhook results.
Challenges
Confidently wrong extractions
The dangerous failure was not a refusal but a plausible wrong number, a rent of $2,450 extracted as $2,540. Log-prob confidence alone missed many of these. Layering deterministic cross-field validation (rent times frequency matching annual figures, dates ordering correctly) plus a cheap second-model agreement check cut undetected critical-field errors to under 0.2%.
Long documents versus context limits
Eighty-page leases exceeded practical context budgets and degraded extraction focus. A retrieval step first located candidate sections per field group using embeddings over document chunks, then extraction ran per section. Accuracy on long documents rose 11 points and token cost per document fell by half.
Evaluation before automation trust
Operations would not turn off manual keying based on a demo. We ran the pipeline in shadow mode against live manual entry for six weeks, building both the golden dataset and a defensible accuracy report per field per document type. The threshold negotiation that followed was grounded in that data, not vibes.