User question
chat
- feeds
- BM25 · Dense search
All case studies
LLMRAGInternal Tools4 months
Internal LLM Copilot (Policies + Tools)
Staff asked colleagues instead of reading the policy. Now they ask and get an answer they can check.
Answer accuracy
89.2%
Hybrid retrieval + reranker + citations
Was
61%
Naive single-vector RAG pilot
In plain terms
What this actually was.
No jargon in this section. The technical write-up is further down.
What was going wrong
Company knowledge sat in policy documents nobody read, so people interrupted colleagues instead. An earlier chatbot attempt made things worse: it answered confidently with no way to tell whether it was right.
What we built
We built an assistant that searches the actual documents, ranks what it finds, and answers only with a link to the source it used — plus a test suite that runs before any change goes live.
What changed
Answers went from 61% correct to 89%, every answer arrives with a source to verify, and new staff get up to speed measurably faster.
Architecture
How it fits together.
Scroll through the stages. Anything marked as added is a component that did not exist before this project.
BM25
lexical
- receives from
- User question
- feeds
- Reranker
Dense search
embeddings
- receives from
- User question
- feeds
- Reranker
Reranker
top-50 → 8
- receives from
- BM25 · Dense search
- feeds
- LLMcontext
LLM
grounded
- receives from
- Rerankercontext
- feeds
- Tool callsif needed · Answer
Tool calls
live state
- receives from
- LLMif needed
Answer
with citations
- receives from
- LLM
- feeds
- Eval harnessscored
Eval harness
340 tasks
- receives from
- Answerscored
User question
chat
- feeds
- BM25 · Dense search
BM25
lexical
- receives from
- User question
- feeds
- Reranker
Dense search
embeddings
- receives from
- User question
- feeds
- Reranker
Reranker
top-50 → 8
We added this- receives from
- BM25 · Dense search
- feeds
- LLMcontext
LLM
grounded
- receives from
- Rerankercontext
- feeds
- Tool callsif needed · Answer
Tool calls
live state
We added this- receives from
- LLMif needed
Answer
with citations
- receives from
- LLM
- feeds
- Eval harnessscored
Eval harness
340 tasks
We added this- receives from
- Answerscored
The write-up
How it was actually built.
Dataset, approach, measured results and the stack. Written for whoever has to review it.
LLMRAGInternal Tools4 months
Internal LLM Copilot (Policies + Tools)
A secure assistant that answers questions and triggers internal tools with guardrails.
Problem
Institutional knowledge lived in scattered policy documents. Staff asked colleagues instead of reading them, and a naive chatbot pilot produced confident answers with no way to check them.
Dataset
- rows
- 128k document chunks
- features
- 1536-dim embeddings
- sources
- Policies, wikis, runbooks, tickets
- window
- Full corpus
Approach
- 01Hybrid retrieval combining BM25 with dense vectors, then a reranker
- 02Answers constrained to retrieved spans with mandatory citations
- 03Tool calling for the lookups that need live system state
- 04A 340-question evaluation harness gating every prompt change
Metrics
- 89.2%
- answer accuracy on eval set
- 100%
- citation coverage span-linked
- 0.94
- retrieval recall@10 +0.21 vs dense
- 2.1s
- response p95 streamed
Business impact
- Policy questions answered in seconds with a source to verify
- Onboarding time for new staff measurably shorter
- Every prompt change now regression-tested before release
Tech stack
- TypeScript
- Next.js
- Python
- OpenAI-compatible API
- PostgreSQL
- LangChain
- Vector DB
Next
Need a system like this?
Answer six questions and we will tell you whether this shape fits your problem — including when it does not.