Teams that want an assistant over their own documents tend to ask the same first question: should we train a model on our documents, or connect a model to them? For a knowledge base the default answer is to connect it, which is what retrieval-augmented generation (RAG) does, and to build a test set of real questions before choosing anything else. This note explains both approaches in plain terms, gives a decision table, and then walks through a two-week test you can run with one engineer and one person who knows the documents.
RAG and fine-tuning in plain terms
A language model knows what was in its training data. It does not know your returns policy, last month's price list or the internal procedure for approving a supplier.
RAG fixes that at question time. When someone asks a question, the system searches your documents, takes the handful of passages that look most relevant, and gives them to the model with an instruction: answer from these passages and say which ones you used. The documents stay where they are. Edit a policy, re-index it, and the next answer reflects the edit. The technique was described in a 2020 research paper that paired a text generator with a searchable document index; the core idea has not changed since.
Fine-tuning continues training a model on examples of inputs and the outputs you want. It changes the model's habits: tone, structure, vocabulary, how it handles one narrow task. It is a weak way to teach facts that change. Every edit to the source needs a new training run, the model cannot point to where an answer came from, and anything it learned is available to everyone who can use it.
Most companies already work this way. Menlo Ventures' December 2025 report on enterprise generative AI found prompt design to be the most common way companies adapt models, with RAG second; fine-tuning and reinforcement learning were still niche and used mainly by frontier teams.
What it costs. The two-week test below costs the time of one engineer and a few days of a subject expert, plus model usage that is small at test-set scale. In production, running cost follows one formula: questions per month × tokens per question × price per token. As an illustration: 10,000 questions a month, each sending five passages of about 400 tokens plus 500 tokens of instructions and question, and getting a 300-token answer, is 10,000 × 2,800 = 28 million tokens. At an illustrative blended price of $5 per million tokens, that is $140 a month. Use your provider's current prices; the formula is the point. If you want the system built rather than tested, our published band for an internal RAG copilot is $8k–18k over 4–8 weeks, and the pricing page shows the rate card behind it.
The decision table
| What you need | RAG | Fine-tuning |
|---|---|---|
| Knowledge that changes weekly or daily | Edits show up after the next re-index | Each change needs a new training run and a new evaluation |
| A source link for every answer | Built in: the answer cites the passages it used | No reliable way to say where an answer came from |
| Different people allowed to see different documents | Passages are filtered by the user's permissions before the model sees them | Whatever is trained in is visible to every user of the model |
| A fixed output format or house tone | Possible with instructions and examples in the prompt, sometimes inconsistent | Good fit |
| Low latency | Adds a search step and a longer prompt | No extra step; a smaller tuned model can also be faster |
| Cost to update | Re-index the changed documents | Prepare examples, retrain, re-evaluate |
| Data needed to start | The documents and a test set of real questions | Example inputs with ideal outputs checked by people, plus the same test set |
Count which rows matter for your case. For a knowledge base, the first three usually decide it: the content changes, people want to check the source, and not everyone may read everything.
Fine-tuning earns its place in a narrower set of cases: answers must always follow a strict structure; a high-volume task such as routing tickets into categories could run on a smaller, cheaper model; or the tone must match a style guide that prompting does not hold. Even then it usually sits on top of retrieval rather than replacing it, as a tuned model that still reads the retrieved passages.
So the default is RAG plus an evaluation set first. Fine-tune later, for format, tone or a narrow task, and only if the test set shows that the remaining errors are of that kind.
The two-week test
The test replaces opinions with a number. At the end you have a score on your own questions and a list of failures sorted by cause, which tells you whether to build, what to fix, and whether fine-tuning would help at all.
Step 1: collect real questions
Take questions people have actually asked: helpdesk tickets, internal chat threads, intranet search logs, the questions the policy owner answers by e-mail. Aim for 150 to 300. Include 10–15% that your documents cannot answer, because declining correctly is part of the job.
Why that many: with 200 questions and a true accuracy around 80%, the 95% margin of error is about ±5.5 percentage points (1.96 × √(0.8 × 0.2 / 200) ≈ 0.055). A smaller set cannot tell two versions of the system apart.
For each question, a person who knows the material writes the expected answer and names the document it should come from. One JSON line per question:
{"id": "q041", "question": "Who approves a new supplier above the purchasing threshold?", "expected": "Procurement lead and finance, both in writing.", "doc": "supplier-onboarding", "groups": ["procurement"], "answerable": true}
Every answer then gets one of four labels:
- correct: right, complete, and cites the right document;
- partly: right but incomplete, or right with a wrong or missing citation;
- wrong: states something false, or answers a question the documents cannot answer;
- declined: says it cannot answer.
Declined is a separate label on purpose. On an answerable question it is a miss, but a cheaper miss than a wrong answer, and the business decides how many of each it can accept. Agree that threshold before the first run, the same way a metric is agreed before a model is trained.
A domain expert grades the first 50 answers by hand. After that, a second model can grade the rest against the expected answers, as long as its labels agree with the expert's on those 50. Both practices are common: in LangChain's survey of 1,340 practitioners building agents, run in November and December 2025, 59.8% used human review, 53.3% used a model as a judge, and 52.4% ran offline evaluations on test sets.
Step 2: index passages with their permissions
Split documents along their headings into passages of a few hundred words. Keep the document id, title, link and access groups with every passage. Permissions belong to the passage from the start; adding them later means re-indexing everything.
Step 3: hybrid retrieval with reciprocal rank fusion
Keyword search (BM25) finds exact terms such as product codes, form numbers and acronyms. Embedding search finds paraphrases: "time off" when the policy says "annual leave". Each misses what the other catches, so use both and merge the rankings.
Reciprocal rank fusion (RRF) merges by position rather than by score, so the two scoring scales never need calibrating: each passage gets 1 / (k + rank) from each list, summed. The 2009 paper that introduced it fixed k at 60 and found that value near-optimal but not critical.
import numpy as np
from rank_bm25 import BM25Okapi
def tokenize(text):
return text.lower().split()
class HybridIndex:
def __init__(self, chunks, embed):
# chunks: dicts with "text", "doc", "groups" (a set); embed: list[str] -> unit vectors
self.chunks, self.embed = chunks, embed
self.bm25 = BM25Okapi([tokenize(c["text"]) for c in chunks])
self.vectors = np.asarray(embed([c["text"] for c in chunks]))
def search(self, query, user_groups, k=5, depth=50, rrf_k=60):
allowed = [i for i, c in enumerate(self.chunks) if c["groups"] & user_groups]
lexical = self.bm25.get_scores(tokenize(query))
semantic = self.vectors @ np.asarray(self.embed([query]))[0]
fused = {}
for scores in (lexical, semantic):
ranked = sorted(allowed, key=lambda i: scores[i], reverse=True)[:depth]
for rank, i in enumerate(ranked, start=1):
fused[i] = fused.get(i, 0.0) + 1.0 / (rrf_k + rank)
top = sorted(fused, key=fused.get, reverse=True)[:k]
return [{**self.chunks[i], "similarity": float(semantic[i])} for i in top]
embed is any embedding function, hosted or open-source; swapping it changes one argument. The permission filter runs before ranking, so a passage the user cannot open never reaches the model.
Step 4: answer with citations, or decline
Two checks keep the system from making things up. Before calling the model: if the best passage is not similar enough to the question, decline without generating anything. After: every answer must cite passages, and every cited number must exist. Anything else counts as declined.
import re
PROMPT = """Answer the question using only the numbered passages below.
After each sentence, cite the passages you used, for example [2].
If the passages do not contain the answer, reply with exactly: NOT_FOUND
{passages}
Question: {question}"""
def answer(question, index, user_groups, llm, min_similarity=0.35):
hits = index.search(question, user_groups)
if not hits or max(h["similarity"] for h in hits) < min_similarity:
return {"status": "declined", "text": None, "docs": []}
passages = "\n\n".join(f"[{n}] ({h['doc']}) {h['text']}" for n, h in enumerate(hits, 1))
text = llm(PROMPT.format(passages=passages, question=question)).strip()
cited = {int(n) for n in re.findall(r"\[(\d+)\]", text)}
if text == "NOT_FOUND" or not cited or not cited <= set(range(1, len(hits) + 1)):
return {"status": "declined", "text": None, "docs": []}
return {"status": "answered", "text": text, "docs": [hits[n - 1]["doc"] for n in sorted(cited)]}
llm is any function that takes a prompt and returns text, so the model can be replaced without touching this code. The 0.35 threshold is a placeholder: similarity scales differ between embedding models. Set it on your test set so that the unanswerable questions are declined without losing too many answerable ones.
Step 5: the scoring loop
import json
from collections import Counter
def run_eval(path, index, llm, judge):
items = [json.loads(line) for line in open(path)]
labels, found = Counter(), 0
for item in items:
groups = set(item["groups"])
if item["answerable"]:
hits = index.search(item["question"], groups)
found += item["doc"] in {h["doc"] for h in hits}
result = answer(item["question"], index, groups, llm)
if not item["answerable"]:
label = "correct" if result["status"] == "declined" else "wrong"
elif result["status"] == "declined":
label = "declined"
else: # judge returns "correct", "partly" or "wrong"
label = judge(item["question"], item["expected"], result["text"])
labels[label] += 1
answerable = sum(1 for item in items if item["answerable"])
print({k: round(v / len(items), 3) for k, v in labels.items()})
print("expected document in top 5:", round(found / answerable, 3))
The second number matters as much as the first. If the right document was not in the top five, the model never had a chance, and the fix is in retrieval or in the documents, not in the model. Save each run's labels with the date and the change that was made, so every change is compared against the same baseline.
Week 1 and week 2
Week 1: the baseline.
- Days 1–2: collect and write the questions with the document owners; agree the acceptance threshold.
- Day 3: index the documents; run keyword search alone, then hybrid, and record the top-5 hit rate for both.
- Days 4–5: first full run; the expert grades 50 answers; check the judge against them; grade the rest.
The output of week 1 is a score and a list of failures sorted into four causes: the document is missing or out of date; retrieval missed the right passage; the model had the right passage and still answered badly; the system should have declined and did not.
Week 2: fix in order of size.
- Start with the largest cause. Missing or stale documents are a content task for their owners. Retrieval misses usually respond to better chunking, document titles and headings added to each passage, or synonyms for internal jargon. Bad answers from good passages respond to prompt changes and, sometimes, a different model.
- Re-run the full set after each change. Keep a change only if the score moves by more than the margin of error.
- Tune the decline threshold last, on the final retrieval setup.
At the end of week 2 the decision is usually clear. If the wrong-answer rate is under the agreed threshold, build the production version and keep the test set as its release gate. If the remaining failures are about format or tone while the facts are right, that is the case where fine-tuning is worth testing, and the test set already exists to show whether it helped. If most failures are missing documents, the problem is the knowledge base, and no choice of model will fix it.
This is the order we follow when building RAG copilots: real questions first, then retrieval, then a score that every release has to beat.
Sources
- Menlo Ventures, 2025: The State of Generative AI in the Enterprise, December 2025 — https://menlovc.com/perspective/2025-the-state-of-generative-ai-in-the-enterprise/
- LangChain, State of Agent Engineering, December 2025 — https://www.langchain.com/state-of-agent-engineering
- NeurIPS 2020 (arXiv preprint), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, May 2020 — https://arxiv.org/abs/2005.11401
- ACM SIGIR 2009, Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods, July 2009 — https://doi.org/10.1145/1571941.1572114