Business Problem
A 1,400-person company had institutional knowledge scattered across a wiki, SharePoint, policy PDFs, product docs, and years of resolved support tickets, around 60k documents in total. Staff either interrupted senior colleagues or filed internal tickets; the internal support team spent an estimated 40% of its time answering questions whose answers already existed somewhere. New-hire ramp time was heavily gated on 'knowing who to ask'.
An off-the-shelf chatbot pilot had failed for two reasons that defined our requirements: it hallucinated policy answers with confident wording, unacceptable when the question is about leave entitlements or security procedure, and it ignored document permissions, happily quoting an executive-only document to anyone. The system had to cite real sources for every claim, respect per-user access at retrieval time, and say 'I do not know' when the corpus did not contain the answer.
Architecture
I built the assistant as a LangGraph agent over a retrieval stack designed permission-first. Documents synced from each source system with their ACLs; chunks were stored in Qdrant with permission metadata, and every retrieval query filtered by the requesting user's resolved group memberships before similarity ranking, so an unauthorized chunk was never even a candidate. Permission filtering at retrieval, not at generation, was the load-bearing decision.
The answer pipeline separated retrieval, synthesis, and verification. A query-planning step decomposed compound questions and rewrote follow-ups using conversation context; hybrid retrieval combined dense vectors with keyword search and a reranker. Synthesis generated answers with inline citations bound to specific chunks, and a verification pass checked each claim against its cited chunk, unsupported claims were removed or the answer downgraded to 'not found in the knowledge base' with suggested contacts.
Every answer shipped with clickable citations opening the source at the relevant section. Feedback (thumbs plus optional correction) flowed into an evaluation set, and a weekly retrieval-quality report drove chunking and reranking tuning. The whole pipeline was traced with OpenTelemetry so any bad answer could be replayed step by step, retrieval candidates, rerank scores, synthesis inputs, verification verdicts.
System Design
Permission filtering applied at retrieval time via ACL metadata in Qdrant, structurally preventing unauthorized content from entering any prompt.
Verification pass binding every claim to a cited chunk, with unsupported claims stripped; groundedness held at 96% on the audited sample.
Hybrid dense-plus-keyword retrieval with reranking, chosen after evaluation showed pure vector search missed exact-term policy queries (form names, system codes) badly.
Explicit 'not found' behavior with routing to the right human team, treated as a first-class answer type rather than a failure.
Incremental source-sync pipeline keeping index freshness under 15 minutes for wiki and ticket updates, with ACL changes propagating on the same path.
Full-pipeline tracing enabling step-level replay of any answer for debugging and audit.
My Responsibilities
Owned architecture and led two engineers; built the retrieval stack and verification pipeline personally.
Designed the permission model mapping four source systems' ACL semantics onto a single retrieval-time filter representation.
Built source connectors and the incremental sync pipeline with ACL propagation.
Created the evaluation framework: a 500-question golden set with per-question expected sources, run against every pipeline change in CI.
Ran the security review with the infosec team, including red-team sessions attempting permission bypass via prompt manipulation.
Drove the phased rollout from a 50-person pilot to company-wide availability, including the feedback triage process.
Implementation
The first two months went into the evaluation set and retrieval quality before any chat UI existed. A 500-question golden set, built with the internal support team from real ticket history, exposed that naive chunking destroyed policy tables and that pure vector search failed exact-code lookups, both fixed before they could become production incidents. Every subsequent change ran against this set in CI with retrieval and groundedness metrics on the PR.
The pilot with 50 support-heavy staff ran six weeks with every answer audited. The audit process directly produced the verification pass design: we categorized every bad answer and found most traced to retrieval gaps, not generation, which redirected effort toward hybrid search and reranking rather than prompt tuning. Groundedness rose from 83% to 96% over the pilot.
Company rollout was deliberately unglamorous: a Next.js chat interface, a Slack entry point, and answer quality that had already been proven. Adoption grew on internal word of mouth to 9k queries a week within four months. The weekly feedback triage, 30 minutes with support leads reviewing flagged answers, remains the engine of continuous quality improvement.
Results
35% of previously ticketed internal questions deflected to self-service answers within four months of company-wide launch.
96% answer groundedness and under 1% audited hallucination rate, sustained across weekly quality reviews.
9k queries per week from 1,100 monthly active staff, with 82% positive feedback rate.
Zero permission violations in production and in 200+ red-team bypass attempts.
New-hire ramp surveys showed 'finding how things work' scores improving 28 points; the assistant became part of onboarding by default.
Internal support team reallocated roughly two full-time-equivalents from repetitive Q&A to process improvement work.
Lessons Learned
Build the evaluation set before the product; every architectural argument afterwards gets settled by numbers instead of opinions.
Permission enforcement belongs at retrieval, structurally, not at generation, behaviorally; the model should never see what the user cannot.
Most 'hallucination' problems in RAG are retrieval problems wearing a generation costume; fix retrieval first.
'I do not know, ask this team' is a feature that builds more trust than a marginal increase in answer coverage.
A standing human feedback loop with the domain team beats any amount of offline benchmark chasing.
Future Improvements
Agentic multi-step retrieval for questions requiring synthesis across many documents, such as policy comparisons.
Proactive stale-answer detection: re-run popular historical queries when their cited sources change and notify prior askers of updates.
Structured action execution, letting the assistant file the correct request form, not just link to it, with human confirmation.
Per-team fine-tuned retrieval profiles, since engineering and HR queries show measurably different retrieval characteristics.
Challenges
Four incompatible permission models
SharePoint, the wiki, the ticket system, and the policy repository each expressed access differently (groups, page inheritance, role tiers, ad hoc shares). I designed a normalized ACL representation with per-source resolvers, synced group membership on a 10-minute cycle, and defaulted to deny on any resolution ambiguity. Red-team sessions produced zero unauthorized retrievals across 200+ bypass attempts.
Hallucination on policy questions
Synthesis occasionally blended two similar policies into a plausible hybrid, the exact failure that killed the previous pilot. The claim-level verification pass, checking each sentence against its cited chunk with a cheap model, plus a rule that policy-category answers must quote rather than paraphrase entitlement numbers, cut audited hallucination below 1% of answers.
Stale knowledge with real consequences
An answer citing a superseded travel policy is worse than no answer. Document-level supersession tracking (new versions demote old chunks), freshness indicators on every citation, and index freshness under 15 minutes addressed the mechanical side; a monthly stale-content report to document owners addressed the human side.