RAG: Retrieval-Augmented Generation, End to End
A model's knowledge is frozen at training time and it has never seen your private documents. Retrieval-augmented generation fixes both by finding relevant passages at question time and putting them in the context, so the answer is grounded and can cite its sources. This piece walks through the whole system — ingestion, chunking, embeddings, vector search, hybrid retrieval, reranking, context assembly, generation and evaluation — and measures a working pipeline on real documentation, including where it fails.
Why RAG
The idea was named by Lewis et al. (2020), who combined a retriever with a generator so the model could condition on retrieved passages. Today "RAG" covers any system that retrieves text and puts it in the prompt.
End-to-End Architecture
A RAG system has two flows that meet at the index:
Data Sources, Ingestion and Chunking
Sources include PDFs, websites, SQL and NoSQL databases, wikis, support tickets, APIs, code repositories and files with images and tables. Ingestion collects them, extracts text and metadata, cleans and normalizes it, and feeds the indexing pipeline. Three decisions here shape everything downstream:
| Step | What good looks like |
|---|---|
| Parsing | Extract text from PDFs, HTML and DOCX; preserve structure (headings, sections, tables); remove boilerplate and navigation. |
| Chunking | Split into passages (often 200–1,000 tokens) with some overlap (for example 50–200 tokens), ideally at section or semantic boundaries rather than mid-sentence. |
| Metadata | Source (file, URL), title, section, page, document type, timestamp, access permissions, custom tags (project, team). |
Chunk size is a real trade-off, and our run shows it. We joined each module's documentation into one long page, split it into chunks of four sizes with 20% overlap, and searched with BM25:
Embeddings and Vector Stores
An embedding model turns text into a vector of numbers — typically hundreds to a few thousand dimensions — positioned so that texts with similar meaning are close together. Common choices include OpenAI's text-embedding-3 models and open models such as BGE and E5. A vector database (Pinecone, Weaviate, Milvus, or a vector extension to an existing database) stores each vector with its original text and metadata, supports fast approximate nearest-neighbour search (indexes such as HNSW or IVF) and metadata filtering, and scales to millions or billions of chunks.
Our script uses LSA (latent semantic analysis): it factorizes the word–document matrix and represents each text as a 128-dimensional vector. LSA predates neural embeddings by decades and captures much less meaning, but it has the same shape — text in, fixed-size vector out, compare by cosine similarity — so every step of the pipeline is real.
Retrieval: Finding Relevant Chunks
At query time the question is embedded with the same model, compared with every chunk vector (cosine similarity or dot product, via an approximate index), and the top-k candidates are returned — often k = 20–50, narrowed later. Metadata filtering restricts the search by source, date, user permissions, project or document type. In our run, "load data from a file" with no filter returned shutil.copyfileobj, statistics.median, os.readv and statistics.variance; filtered to the json module it returned exactly json.load and json.loads. Permissions belong in this filter, applied during the search, so a user can never retrieve a document they aren't allowed to read.
Hybrid Retrieval and Reranking
Hybrid retrieval combines semantic (vector) search with keyword search such as BM25, which rewards exact term matches and is strong on names, IDs, error codes and jargon that embeddings blur. A simple, robust way to merge the two ranked lists is reciprocal rank fusion (RRF): each document scores the sum of 1/(60 + rank) across the lists.
Reranking reorders the top candidates with a more expensive model that reads the query and each passage together — a cross-encoder such as BGE-reranker or Cohere Rerank — so it can judge relevance more precisely than a vector comparison. It's slower, so it runs only on the few dozen candidates retrieval returns.
Measured Results
On 30 questions over 622 documents (each question has one to three correct functions), with three standard metrics — Recall@5 (was a correct document in the top 5?), MRR (how high was the first correct one?) and nDCG@10 (were the correct ones near the top?):
| Method | Recall@5 | MRR@10 | nDCG@10 |
|---|---|---|---|
| BM25 (keyword) | 70% | 0.55 | 0.58 |
| dense (LSA vectors) | 63% | 0.38 | 0.47 |
| hybrid (BM25 + dense, RRF) | 73% | 0.52 | 0.57 |
| hybrid + rerank | 67% | 0.45 | 0.49 |
Three honest lessons:
os.times_result). A reranker is only as good as its model — which is why production systems use trained ones and measure them before shipping.Hybrid retrieval still missed 8 of 30 questions — for example "get the folder my program is currently running in" (the documentation says "current working directory") and "remember results of an expensive function call" (it says "cache"). That's the vocabulary-mismatch problem, and it's exactly where neural embeddings and query rewriting earn their place: rewriting the second question as "cache decorator for function results" put functools.lru_cache first.
Context Assembly, Generation and Grounding
The final prompt combines system instructions ("use the provided documents, cite them"), relevant conversation history, the top-ranked passages with citation markers, any tool definitions, and the user's message — clearly delimited so the model can tell documents from instructions (see Context Engineering). Our run's prompt, after query rewriting:
<system>Answer only from the documents. Cite them as [n].</system> <documents> [1] (functools.lru_cache) functools.lru_cache: Least-recently-used cache decorator. If *maxsize* is set to None, the LRU features are disabled and the cache can grow without bo... [2] (functools.recursive_repr) functools.recursive_repr: Decorator to make a repr function return fillvalue for a recursive call... [3] (functools.cache) functools.cache: Simple lightweight unbounded cache. Sometimes called "memoize".... </documents> <user>how do I remember results of an expensive function call</user>
The model then writes an answer grounded in those passages, with citations back to them. Our script has no language model, so it "generates" extractively — it quotes the best-supported sentence from the top document and cites it — and then checks that the quoted text really appears in the cited source: Least-recently-used cache decorator. [1]. A real model paraphrases and synthesizes across passages, which is more useful but means groundedness has to be checked rather than assumed.
Evaluation Metrics
Retrieval metrics need a labelled set of questions with known relevant documents, like our 30. Answer metrics are judged by people or by a model acting as judge; frameworks such as RAGAS and TruLens automate the latter. Evaluate the two halves separately: a wrong answer from a correct retrieval is a generation problem, and a wrong answer from a bad retrieval is a search problem.
Common Failure Modes
| Failure | What to do |
|---|---|
| Poor chunking loses important context | Chunk at section boundaries, add overlap, keep headings with their text |
| Irrelevant or low-quality retrieval | Hybrid search, reranking, better embeddings, evaluate with labelled queries |
| Stale or outdated documents | Re-index on change, timestamp metadata, prefer newer versions |
| Missing documents | Monitor questions that retrieve nothing relevant; fix ingestion gaps |
| Too many irrelevant chunks (context dilution) | Smaller top-k after reranking; relevance thresholds |
| Incorrect metadata or permissions | Validate metadata at ingestion; enforce permissions in the search filter |
| Hallucination despite retrieval | Instruct "answer only from documents", require citations, check groundedness |
| Poor query formulation (vocabulary mismatch) | Query rewriting, multi-query retrieval, neural embeddings |
| Large tool outputs crowding the context | Summarize or truncate tool results before adding them |
| Instructions hidden in retrieved documents | Treat retrieved text as data, delimit it, and keep actions behind permissions (see prompt injection) |
Advanced RAG Techniques
| Technique | Idea |
|---|---|
| Query rewriting | Reformulate the question for better retrieval — as in our "cache decorator" example. HyDE goes further: generate a hypothetical answer and search with that. |
| Multi-query retrieval | Generate several phrasings, retrieve for each, merge the results. |
| Hierarchical / parent–child chunks | Search small chunks for precision, then return their larger parent section for context — resolving our chunk-size trade-off. |
| Graph retrieval | Use a knowledge graph built from the documents for questions about relationships and themes (e.g. Microsoft's GraphRAG). |
| Metadata-aware retrieval | Use filters and structured metadata (time, source, owner) as part of the query. |
| Iterative / agentic retrieval | Let the model decide it needs more information and retrieve again, possibly with a new query. |
| Domain-specific RAG | Custom embeddings, chunking and ranking tuned for one domain (legal, code, medical). |
RAG vs Fine-Tuning vs Tool Use
| Aspect | RAG | Fine-tuning | Tool use |
|---|---|---|---|
| Knowledge source | External documents | The model's weights | External systems (APIs, databases) |
| Best for | Up-to-date or private facts | Changing behaviour, format or style | Actions and live data |
| Requires training | No | Yes | No |
| Data freshness | As fresh as the index | Fixed at training time | Real time |
| Typical uses | Docs, manuals, policies, private knowledge | Domain adaptation, a specific style | Search, calculations, external APIs |
They combine: a fine-tuned model can be better at using retrieved context, and retrieval can itself be a tool the model calls when it decides it needs information. See How Fine-Tuning Works for the middle column.
Run the Pipeline
One script; needs Python 3 and numpy; runs in about a second. It indexes the documentation of 30 standard-library modules on your machine, so the corpus size varies slightly with your Python version.
"""A RAG pipeline, measured on real documents: Python standard-library documentation. Indexing: collect ~700 function docstrings -> clean -> chunk -> index (BM25 keyword index + dense vectors). Query: 30 hand-written questions, each with the function(s) that answer it -> retrieve -> fuse -> rerank -> build a cited prompt -> extractive grounded answer. Metrics: Recall@k, MRR, nDCG. Dense vectors here are LSA (SVD of TF-IDF), a classical stand-in for neural embeddings; it needs only numpy. """ import importlib, inspect, math, re from collections import Counter import numpy as np # ---------------- 1. ingest: collect, extract, clean ---------------- MODULES = ["os", "os.path", "shutil", "json", "re", "math", "statistics", "random", "time", "string", "itertools", "functools", "collections", "subprocess", "glob", "csv", "hashlib", "base64", "textwrap", "heapq", "bisect", "urllib.parse", "zipfile", "tempfile", "uuid", "copy", "difflib", "gzip", "datetime", "fnmatch"] docs = [] # (id, module, text) for mod in MODULES: m = importlib.import_module(mod) for name in sorted(dir(m)): if name.startswith("_"): continue obj = getattr(m, name, None) doc = inspect.getdoc(obj) if (callable(obj) and getattr(obj, "__doc__", None)) else None if not doc or len(doc) < 20: continue doc = re.sub(r"\s+", " ", doc)[:900] # clean: collapse whitespace, cap length docs.append((f"{mod}.{name}", mod, f"{mod}.{name}: {doc}")) print(f"corpus: {len(docs)} documents from {len(MODULES)} modules") def words(t): return re.findall(r"[a-z0-9]+", t.lower()) STOP = set("a an and are as at be by can do does for from how i if in into is it its of on or that the this to " "what with my me get make".split()) def terms(t): # tokens, plus light stemming so "files" matches "file" return [w[:-1] if w.endswith("s") and len(w) > 3 else w for w in words(t) if w not in STOP] # ---------------- 2. index: BM25 and dense (LSA) vectors ---------------- class BM25: def __init__(self, texts, k1=1.5, b=0.75): self.t = [terms(x) for x in texts]; self.N = len(texts); self.k1, self.b = k1, b self.avg = sum(map(len, self.t)) / self.N; self.df = Counter(w for d in self.t for w in set(d)) self.tf = [Counter(d) for d in self.t] def scores(self, q): qt = terms(q); out = np.zeros(self.N) for i, tf in enumerate(self.tf): L = len(self.t[i]) for w in qt: if w in tf: idf = math.log(1 + (self.N - self.df[w] + 0.5) / (self.df[w] + 0.5)) out[i] += idf * tf[w] * (self.k1 + 1) / (tf[w] + self.k1 * (1 - self.b + self.b * L / self.avg)) return out class LSA: def __init__(self, texts, dim=128): self.vocab = {w: j for j, w in enumerate(sorted({w for x in texts for w in terms(x)}))} X = self._tf(texts); df = (X > 0).sum(0); self.idf = np.log(len(texts) / (1 + df)) + 1 X = X * self.idf; U, S, Vt = np.linalg.svd(X, full_matrices=False) self.P = Vt[:dim].T # vocabulary -> dim-dimensional "embedding" space self.E = self._norm(X @ self.P) def _tf(self, texts): X = np.zeros((len(texts), len(self.vocab))) for i, x in enumerate(texts): for w, c in Counter(terms(x)).items(): if w in self.vocab: X[i, self.vocab[w]] = 1 + math.log(c) return X def _norm(self, M): return M / (np.linalg.norm(M, axis=1, keepdims=True) + 1e-9) def embed(self, q): return self._norm((self._tf([q]) * self.idf) @ self.P)[0] def scores(self, q): return self.E @ self.embed(q) # cosine similarity texts = [d[2] for d in docs] bm25, lsa = BM25(texts), LSA(texts) print(f"index: BM25 over {len(bm25.df):,} terms; dense vectors of {lsa.P.shape[1]} dimensions") def rrf(*rankings, k=60): # reciprocal-rank fusion (hybrid retrieval) s = Counter() for r in rankings: for rank, i in enumerate(r): s[i] += 1 / (k + rank + 1) return [i for i, _ in s.most_common()] def rerank(q, cand): # feature reranker: stand-in for a cross-encoder qt = set(terms(q)) def score(i): name = set(terms(docs[i][0].replace(".", " ").replace("_", " "))) first = set(terms(docs[i][2].split(":", 1)[1].split(".")[0])) # the docstring's summary sentence return 2.0 * len(qt & name) + 1.0 * len(qt & first) + 0.5 * bm_cache[i] / (bm_cache.max() + 1e-9) return sorted(cand, key=score, reverse=True) # ---------------- 3. evaluation set: 30 questions with known answers ---------------- QA = [("get the folder my program is currently running in", ["os.getcwd"]), ("delete a folder and everything inside it", ["shutil.rmtree"]), ("copy a file to another location", ["shutil.copy", "shutil.copyfile", "shutil.copy2"]), ("turn a python dictionary into a JSON string", ["json.dumps"]), ("parse JSON text into python objects", ["json.loads"]), ("find every match of a regular expression in a string", ["re.findall", "re.finditer"]), ("replace text that matches a pattern", ["re.sub", "re.subn"]), ("square root of a number", ["math.sqrt", "math.isqrt"]), ("middle value of a list of numbers", ["statistics.median"]), ("pick a random element from a list", ["random.choice"]), ("randomly reorder a list in place", ["random.shuffle"]), ("pause the program for a few seconds", ["time.sleep"]), ("all possible orderings of some items", ["itertools.permutations"]), ("remember results of an expensive function call", ["functools.lru_cache", "functools.cache"]), ("count how many times each item appears", ["collections.Counter"]), ("run a shell command and capture its output", ["subprocess.run", "subprocess.check_output", "subprocess.getoutput"]), ("find files whose names match a wildcard like *.txt", ["glob.glob", "glob.iglob", "fnmatch.filter"]), ("compute a SHA-256 digest", ["hashlib.sha256"]), ("encode bytes as base64", ["base64.b64encode", "base64.standard_b64encode"]), ("wrap a long paragraph to a fixed line width", ["textwrap.wrap", "textwrap.fill"]), ("the n smallest items in a collection", ["heapq.nsmallest"]), ("insert into a sorted list and keep it sorted", ["bisect.insort", "bisect.insort_right", "bisect.insort_left"]), ("split a URL into scheme, host and path", ["urllib.parse.urlparse", "urllib.parse.urlsplit"]), ("create a temporary directory", ["tempfile.mkdtemp", "tempfile.TemporaryDirectory"]), ("generate a random unique identifier", ["uuid.uuid4"]), ("duplicate an object including everything it contains", ["copy.deepcopy"]), ("how similar are two strings", ["difflib.SequenceMatcher", "difflib.get_close_matches"]), ("check whether a path exists on disk", ["os.path.exists", "os.path.lexists"]), ("list the entries in a directory", ["os.listdir", "os.scandir"]), ("combine path components into one path", ["os.path.join"])] ids = [d[0] for d in docs] QA = [(q, [a for a in ans if a in ids]) for q, ans in QA] assert all(a for _, a in QA), "every question needs at least one answer in the corpus" def metrics(rank_fn): rec5 = mrr = ndcg = 0.0 for q, ans in QA: r = [ids[i] for i in rank_fn(q)[:10]] rel = [1 if x in ans else 0 for x in r] rec5 += any(rel[:5]); mrr += next((1 / (k + 1) for k, v in enumerate(rel) if v), 0) dcg = sum(v / math.log2(k + 2) for k, v in enumerate(rel)) ndcg += dcg / sum(1 / math.log2(k + 2) for k in range(min(len(ans), 10))) n = len(QA); return rec5 / n, mrr / n, ndcg / n bm_cache = None def run_bm25(q): return list(np.argsort(-bm25.scores(q))) def run_dense(q): return list(np.argsort(-lsa.scores(q))) def run_hybrid(q): return rrf(run_bm25(q)[:50], run_dense(q)[:50]) def run_hybrid_rerank(q): global bm_cache; bm_cache = bm25.scores(q); c = run_hybrid(q)[:30] return rerank(q, c) + c[30:] print(f"\nretrieval on {len(QA)} questions Recall@5 MRR@10 nDCG@10") for name, fn in [("BM25 (keyword)", run_bm25), ("dense (LSA vectors)", run_dense), ("hybrid (BM25 + dense, RRF)", run_hybrid), ("hybrid + rerank", run_hybrid_rerank)]: r, m, n = metrics(fn); print(f" {name:<30} {r:>6.0%} {m:>6.2f} {n:>6.2f}") miss = [q for q, ans in QA if not set(ids[i] for i in run_hybrid(q)[:5]) & set(ans)] print(f" hybrid misses ({len(miss)}): " + "; ".join(miss)) # ---------------- 4. chunking: size vs recall vs context cost ---------------- by_mod = {} for i, (fid, mod, t) in enumerate(docs): by_mod.setdefault(mod, []).append(t) print("\nchunking (module pages split into chunks, BM25, top 5) Recall@5 tokens in top-5 context") for size in (40, 120, 400, 1500): chunks = [] for mod, ts in by_mod.items(): toks = " ".join(ts).split(); step = max(1, int(size * 0.8)) # 20% overlap for s in range(0, len(toks), step): chunks.append(" ".join(toks[s:s + size])) idx = BM25(chunks); hit = 0; ctx = 0 for q, ans in QA: top = list(np.argsort(-idx.scores(q))[:5]) hit += any(any(a + ":" in chunks[i] for a in ans) for i in top); ctx += sum(len(chunks[i].split()) for i in top) print(f" {size:>5}-word chunks ({len(chunks):>5} chunks) {hit / len(QA):>5.0%} {ctx / len(QA):>8,.0f}") # ---------------- 5. metadata filtering ---------------- q = "load data from a file" print(f"\nmetadata filter - query: {q!r}") print(" no filter: ", [ids[i] for i in run_hybrid(q)[:4]]) print(" module == json:", [ids[i] for i in run_hybrid(q) if docs[i][1] == "json"][:4]) # ---------------- 6. query rewriting, context assembly, grounded cited answer ---------------- q = "how do I remember results of an expensive function call" print(f"\nquery: {q!r}\n hybrid top 3: {[ids[i] for i in run_hybrid(q)[:3]]} <- vocabulary mismatch: the docs say 'cache', not 'remember'") rq = "cache decorator for function results" # written by hand here; in production an LLM rewrites the query top = run_hybrid(rq)[:3] print(f" rewritten as {rq!r}\n hybrid top 3: {[ids[i] for i in top]}") prompt = "<system>Answer only from the documents. Cite them as [n].</system>\n<documents>\n" + "\n".join( f"[{n + 1}] ({ids[i]}) {docs[i][2][:150]}..." for n, i in enumerate(top)) + f"\n</documents>\n<user>{q}</user>" print(f"\nprompt to the model ({len(prompt.split())} words):\n" + prompt) qt = set(terms(q + " " + rq)); best = None for n, i in enumerate(top[:1]): # extractive "generation": the best-supported sentence of the for s in re.split(r"(?<=[.!?])\s+", docs[i][2].split(":", 1)[1]): # top-ranked document, with its citation sc = len(qt & set(terms(s))) if best is None or sc > best[0]: best = (sc, s.strip(), n + 1) answer = f"{best[1]} [{best[2]}]" print(f"\nanswer: {answer}\ngrounded in the cited document: {best[1] in docs[top[best[2] - 1]][2]}")
Output (Python 3.11, numpy 2.x; deterministic):
corpus: 622 documents from 30 modules
index: BM25 over 2,486 terms; dense vectors of 128 dimensions
retrieval on 30 questions Recall@5 MRR@10 nDCG@10
BM25 (keyword) 70% 0.55 0.58
dense (LSA vectors) 63% 0.38 0.47
hybrid (BM25 + dense, RRF) 73% 0.52 0.57
hybrid + rerank 67% 0.45 0.49
hybrid misses (8): get the folder my program is currently running in; delete a folder and everything inside it; find every match of a regular expression in a string; replace text that matches a pattern; all possible orderings of some items; remember results of an expensive function call; compute a SHA-256 digest; duplicate an object including everything it contains
chunking (module pages split into chunks, BM25, top 5) Recall@5 tokens in top-5 context
40-word chunks ( 775 chunks) 63% 197
120-word chunks ( 267 chunks) 77% 571
400-word chunks ( 90 chunks) 87% 1,676
1500-word chunks ( 37 chunks) 90% 4,164
metadata filter - query: 'load data from a file'
no filter: ['shutil.copyfileobj', 'statistics.median', 'os.readv', 'statistics.variance']
module == json: ['json.load', 'json.loads']
query: 'how do I remember results of an expensive function call'
hybrid top 3: ['os.getpgid', 'os.getsid', 'functools.recursive_repr'] <- vocabulary mismatch: the docs say 'cache', not 'remember'
rewritten as 'cache decorator for function results'
hybrid top 3: ['functools.lru_cache', 'functools.recursive_repr', 'functools.cache']
prompt to the model (70 words):
<system>Answer only from the documents. Cite them as [n].</system>
<documents>
[1] (functools.lru_cache) functools.lru_cache: Least-recently-used cache decorator. If *maxsize* is set to None, the LRU features are disabled and the cache can grow without bo...
[2] (functools.recursive_repr) functools.recursive_repr: Decorator to make a repr function return fillvalue for a recursive call...
[3] (functools.cache) functools.cache: Simple lightweight unbounded cache. Sometimes called "memoize"....
</documents>
<user>how do I remember results of an expensive function call</user>
answer: Least-recently-used cache decorator. [1]
grounded in the cited document: True⚠️ Notes and Corrections
- Embedding size: it pairs "text-embedding-3-large" with "768 or 1,536 dims". OpenAI's text-embedding-3-small produces 1,536 dimensions and text-embedding-3-large 3,072 by default (both can be shortened); 768 is typical of BERT-sized open models.
- "OpenAI rerankers": we couldn't confirm a standalone OpenAI reranking model; OpenAI's file-search tool applies its own ranking internally. BGE-reranker and Cohere Rerank are standalone rerankers.
- Filtering order: the retrieval panel shows metadata filtering after the top-k search. Filtering after the fact can leave too few results, and for permissions it's unsafe as a design; filters are usually applied during the search.
- The example answer and query trace are illustrative, not the output of a real system.