Home › Blog › RAG: Retrieval-Augmented Generation
Deep Dive · With a measured RAG pipeline 🔎

RAG: Retrieval-Augmented Generation, End to End

A model's knowledge is frozen at training time and it has never seen your private documents. Retrieval-augmented generation fixes both by finding relevant passages at question time and putting them in the context, so the answer is grounded and can cite its sources. This piece walks through the whole system — ingestion, chunking, embeddings, vector search, hybrid retrieval, reranking, context assembly, generation and evaluation — and measures a working pipeline on real documentation, including where it fails.

How to read this: "Our run" numbers come from the script in Run the Pipeline, which indexes 622 real documents (Python standard-library documentation) and evaluates retrieval on 30 questions I wrote, each with a known correct answer. Its dense vectors are LSA, a classical technique, because no neural embedding model could be downloaded in the environment used; neural embeddings would do better on paraphrased questions. This is the ninth piece in the How LLMs Are Built series, following Context Engineering and Memory.

Why RAG

1A model's knowledge is frozen at training time; it doesn't know recent events.
2It can't see private or internal data — your policies, tickets, code or contracts.
3It may not know domain specifics well, and when it doesn't, it may guess.
4RAG gives it external, up-to-date, trusted information at inference time, which reduces hallucination and makes citations possible.

The idea was named by Lewis et al. (2020), who combined a retriever with a generator so the model could condition on retrieved passages. Today "RAG" covers any system that retrieves text and puts it in the prompt.

End-to-End Architecture

A RAG system has two flows that meet at the index:

A · Offline indexing (prepare the data)
Documents (PDFs, websites, databases, wikis, tickets) → parse and clean (extract text, remove noise, keep structure) → chunk into passages → embed each chunk as a vector → store vectors, text and metadata in a vector database.
B · Online query (answer a question)
User question → embed the query → retrieve (similarity search plus filters) → rerank → build the context (question, passages, instructions) → LLM generation → a grounded answer with citations.

Data Sources, Ingestion and Chunking

Sources include PDFs, websites, SQL and NoSQL databases, wikis, support tickets, APIs, code repositories and files with images and tables. Ingestion collects them, extracts text and metadata, cleans and normalizes it, and feeds the indexing pipeline. Three decisions here shape everything downstream:

StepWhat good looks like
ParsingExtract text from PDFs, HTML and DOCX; preserve structure (headings, sections, tables); remove boilerplate and navigation.
ChunkingSplit into passages (often 200–1,000 tokens) with some overlap (for example 50–200 tokens), ideally at section or semantic boundaries rather than mid-sentence.
MetadataSource (file, URL), title, section, page, document type, timestamp, access permissions, custom tags (project, team).

Chunk size is a real trade-off, and our run shows it. We joined each module's documentation into one long page, split it into chunks of four sizes with 20% overlap, and searched with BM25:

Recall@5Tokens in top-5 context 40 words 63% 197 120 words 77% 571 400 words 87% 1,676 1,500 words 90% 4,164
Our run: bigger chunks find the answer more often (the relevant text is more likely to be inside a retrieved chunk), but the five retrieved chunks cost far more context tokens — 21× more at 1,500 words than at 40. Somewhere in the middle is usually right; where depends on your documents.

Embeddings and Vector Stores

An embedding model turns text into a vector of numbers — typically hundreds to a few thousand dimensions — positioned so that texts with similar meaning are close together. Common choices include OpenAI's text-embedding-3 models and open models such as BGE and E5. A vector database (Pinecone, Weaviate, Milvus, or a vector extension to an existing database) stores each vector with its original text and metadata, supports fast approximate nearest-neighbour search (indexes such as HNSW or IVF) and metadata filtering, and scales to millions or billions of chunks.

Our script uses LSA (latent semantic analysis): it factorizes the word–document matrix and represents each text as a 128-dimensional vector. LSA predates neural embeddings by decades and captures much less meaning, but it has the same shape — text in, fixed-size vector out, compare by cosine similarity — so every step of the pipeline is real.

Retrieval: Finding Relevant Chunks

At query time the question is embedded with the same model, compared with every chunk vector (cosine similarity or dot product, via an approximate index), and the top-k candidates are returned — often k = 20–50, narrowed later. Metadata filtering restricts the search by source, date, user permissions, project or document type. In our run, "load data from a file" with no filter returned shutil.copyfileobj, statistics.median, os.readv and statistics.variance; filtered to the json module it returned exactly json.load and json.loads. Permissions belong in this filter, applied during the search, so a user can never retrieve a document they aren't allowed to read.

Hybrid Retrieval and Reranking

Hybrid retrieval combines semantic (vector) search with keyword search such as BM25, which rewards exact term matches and is strong on names, IDs, error codes and jargon that embeddings blur. A simple, robust way to merge the two ranked lists is reciprocal rank fusion (RRF): each document scores the sum of 1/(60 + rank) across the lists.

Reranking reorders the top candidates with a more expensive model that reads the query and each passage together — a cross-encoder such as BGE-reranker or Cohere Rerank — so it can judge relevance more precisely than a vector comparison. It's slower, so it runs only on the few dozen candidates retrieval returns.

Measured Results

On 30 questions over 622 documents (each question has one to three correct functions), with three standard metrics — Recall@5 (was a correct document in the top 5?), MRR (how high was the first correct one?) and nDCG@10 (were the correct ones near the top?):

MethodRecall@5MRR@10nDCG@10
BM25 (keyword)70%0.550.58
dense (LSA vectors)63%0.380.47
hybrid (BM25 + dense, RRF)73%0.520.57
hybrid + rerank67%0.450.49

Three honest lessons:

1Hybrid had the best recall, finding a correct document in the top 5 more often than either method alone — the usual reason to use it — though on this small set BM25 alone still ranked the first hit slightly higher (MRR).
2Our LSA vectors trailed BM25. Documentation is full of exact names, which keyword search handles well, and LSA is a weak stand-in for a neural embedding model.
3Our reranker made things worse. It's a hand-written scoring rule, not a trained cross-encoder, and it latched onto surface matches (the word "results" pulled up os.times_result). A reranker is only as good as its model — which is why production systems use trained ones and measure them before shipping.

Hybrid retrieval still missed 8 of 30 questions — for example "get the folder my program is currently running in" (the documentation says "current working directory") and "remember results of an expensive function call" (it says "cache"). That's the vocabulary-mismatch problem, and it's exactly where neural embeddings and query rewriting earn their place: rewriting the second question as "cache decorator for function results" put functools.lru_cache first.

Context Assembly, Generation and Grounding

The final prompt combines system instructions ("use the provided documents, cite them"), relevant conversation history, the top-ranked passages with citation markers, any tool definitions, and the user's message — clearly delimited so the model can tell documents from instructions (see Context Engineering). Our run's prompt, after query rewriting:

<system>Answer only from the documents. Cite them as [n].</system>
<documents>
[1] (functools.lru_cache) functools.lru_cache: Least-recently-used cache decorator. If *maxsize* is set to None, the LRU features are disabled and the cache can grow without bo...
[2] (functools.recursive_repr) functools.recursive_repr: Decorator to make a repr function return fillvalue for a recursive call...
[3] (functools.cache) functools.cache: Simple lightweight unbounded cache. Sometimes called "memoize"....
</documents>
<user>how do I remember results of an expensive function call</user>

The model then writes an answer grounded in those passages, with citations back to them. Our script has no language model, so it "generates" extractively — it quotes the best-supported sentence from the top document and cites it — and then checks that the quoted text really appears in the cited source: Least-recently-used cache decorator. [1]. A real model paraphrases and synthesizes across passages, which is more useful but means groundedness has to be checked rather than assumed.

Evaluation Metrics

Retrieval quality
Recall@k — did we retrieve the relevant chunks? Precision@k — how many retrieved chunks are relevant? MRR — mean reciprocal rank of the first relevant chunk. nDCG — ranking quality, rewarding relevant chunks near the top.
Answer quality
Faithfulness — no claims beyond the evidence. Groundedness — supported by retrieved text. Citation accuracy — citations point to the right sources. Helpfulness — does it answer the question?

Retrieval metrics need a labelled set of questions with known relevant documents, like our 30. Answer metrics are judged by people or by a model acting as judge; frameworks such as RAGAS and TruLens automate the latter. Evaluate the two halves separately: a wrong answer from a correct retrieval is a generation problem, and a wrong answer from a bad retrieval is a search problem.

Common Failure Modes

FailureWhat to do
Poor chunking loses important contextChunk at section boundaries, add overlap, keep headings with their text
Irrelevant or low-quality retrievalHybrid search, reranking, better embeddings, evaluate with labelled queries
Stale or outdated documentsRe-index on change, timestamp metadata, prefer newer versions
Missing documentsMonitor questions that retrieve nothing relevant; fix ingestion gaps
Too many irrelevant chunks (context dilution)Smaller top-k after reranking; relevance thresholds
Incorrect metadata or permissionsValidate metadata at ingestion; enforce permissions in the search filter
Hallucination despite retrievalInstruct "answer only from documents", require citations, check groundedness
Poor query formulation (vocabulary mismatch)Query rewriting, multi-query retrieval, neural embeddings
Large tool outputs crowding the contextSummarize or truncate tool results before adding them
Instructions hidden in retrieved documentsTreat retrieved text as data, delimit it, and keep actions behind permissions (see prompt injection)

Advanced RAG Techniques

TechniqueIdea
Query rewritingReformulate the question for better retrieval — as in our "cache decorator" example. HyDE goes further: generate a hypothetical answer and search with that.
Multi-query retrievalGenerate several phrasings, retrieve for each, merge the results.
Hierarchical / parent–child chunksSearch small chunks for precision, then return their larger parent section for context — resolving our chunk-size trade-off.
Graph retrievalUse a knowledge graph built from the documents for questions about relationships and themes (e.g. Microsoft's GraphRAG).
Metadata-aware retrievalUse filters and structured metadata (time, source, owner) as part of the query.
Iterative / agentic retrievalLet the model decide it needs more information and retrieve again, possibly with a new query.
Domain-specific RAGCustom embeddings, chunking and ranking tuned for one domain (legal, code, medical).

RAG vs Fine-Tuning vs Tool Use

AspectRAGFine-tuningTool use
Knowledge sourceExternal documentsThe model's weightsExternal systems (APIs, databases)
Best forUp-to-date or private factsChanging behaviour, format or styleActions and live data
Requires trainingNoYesNo
Data freshnessAs fresh as the indexFixed at training timeReal time
Typical usesDocs, manuals, policies, private knowledgeDomain adaptation, a specific styleSearch, calculations, external APIs

They combine: a fine-tuned model can be better at using retrieved context, and retrieval can itself be a tool the model calls when it decides it needs information. See How Fine-Tuning Works for the middle column.

RAG in one paragraph
Offline, documents are parsed, cleaned, chunked, embedded and stored with metadata in a vector index. Online, the question is embedded and matched against that index — ideally combined with keyword search, filtered by metadata and permissions, and reranked by a trained model — and the best passages are placed in a delimited prompt that tells the model to answer from them and cite them. Chunk size trades recall against context cost, vocabulary mismatch is the most common retrieval failure, and every stage should be measured: retrieval with recall and ranking metrics on labelled questions, answers with faithfulness and citation accuracy.

Run the Pipeline

One script; needs Python 3 and numpy; runs in about a second. It indexes the documentation of 30 standard-library modules on your machine, so the corpus size varies slightly with your Python version.

"""A RAG pipeline, measured on real documents: Python standard-library documentation.

Indexing: collect ~700 function docstrings -> clean -> chunk -> index (BM25 keyword index + dense vectors).
Query: 30 hand-written questions, each with the function(s) that answer it -> retrieve -> fuse -> rerank
-> build a cited prompt -> extractive grounded answer. Metrics: Recall@k, MRR, nDCG.
Dense vectors here are LSA (SVD of TF-IDF), a classical stand-in for neural embeddings; it needs only numpy.
"""
import importlib, inspect, math, re
from collections import Counter
import numpy as np

# ---------------- 1. ingest: collect, extract, clean ----------------
MODULES = ["os", "os.path", "shutil", "json", "re", "math", "statistics", "random", "time", "string", "itertools",
           "functools", "collections", "subprocess", "glob", "csv", "hashlib", "base64", "textwrap", "heapq",
           "bisect", "urllib.parse", "zipfile", "tempfile", "uuid", "copy", "difflib", "gzip", "datetime", "fnmatch"]
docs = []                                            # (id, module, text)
for mod in MODULES:
    m = importlib.import_module(mod)
    for name in sorted(dir(m)):
        if name.startswith("_"): continue
        obj = getattr(m, name, None)
        doc = inspect.getdoc(obj) if (callable(obj) and getattr(obj, "__doc__", None)) else None
        if not doc or len(doc) < 20: continue
        doc = re.sub(r"\s+", " ", doc)[:900]         # clean: collapse whitespace, cap length
        docs.append((f"{mod}.{name}", mod, f"{mod}.{name}: {doc}"))
print(f"corpus: {len(docs)} documents from {len(MODULES)} modules")

def words(t): return re.findall(r"[a-z0-9]+", t.lower())
STOP = set("a an and are as at be by can do does for from how i if in into is it its of on or that the this to "
           "what with my me get make".split())
def terms(t):                                        # tokens, plus light stemming so "files" matches "file"
    return [w[:-1] if w.endswith("s") and len(w) > 3 else w for w in words(t) if w not in STOP]

# ---------------- 2. index: BM25 and dense (LSA) vectors ----------------
class BM25:
    def __init__(self, texts, k1=1.5, b=0.75):
        self.t = [terms(x) for x in texts]; self.N = len(texts); self.k1, self.b = k1, b
        self.avg = sum(map(len, self.t)) / self.N; self.df = Counter(w for d in self.t for w in set(d))
        self.tf = [Counter(d) for d in self.t]
    def scores(self, q):
        qt = terms(q); out = np.zeros(self.N)
        for i, tf in enumerate(self.tf):
            L = len(self.t[i])
            for w in qt:
                if w in tf:
                    idf = math.log(1 + (self.N - self.df[w] + 0.5) / (self.df[w] + 0.5))
                    out[i] += idf * tf[w] * (self.k1 + 1) / (tf[w] + self.k1 * (1 - self.b + self.b * L / self.avg))
        return out
class LSA:
    def __init__(self, texts, dim=128):
        self.vocab = {w: j for j, w in enumerate(sorted({w for x in texts for w in terms(x)}))}
        X = self._tf(texts); df = (X > 0).sum(0); self.idf = np.log(len(texts) / (1 + df)) + 1
        X = X * self.idf; U, S, Vt = np.linalg.svd(X, full_matrices=False)
        self.P = Vt[:dim].T                          # vocabulary -> dim-dimensional "embedding" space
        self.E = self._norm(X @ self.P)
    def _tf(self, texts):
        X = np.zeros((len(texts), len(self.vocab)))
        for i, x in enumerate(texts):
            for w, c in Counter(terms(x)).items():
                if w in self.vocab: X[i, self.vocab[w]] = 1 + math.log(c)
        return X
    def _norm(self, M): return M / (np.linalg.norm(M, axis=1, keepdims=True) + 1e-9)
    def embed(self, q): return self._norm((self._tf([q]) * self.idf) @ self.P)[0]
    def scores(self, q): return self.E @ self.embed(q)          # cosine similarity
texts = [d[2] for d in docs]
bm25, lsa = BM25(texts), LSA(texts)
print(f"index: BM25 over {len(bm25.df):,} terms; dense vectors of {lsa.P.shape[1]} dimensions")

def rrf(*rankings, k=60):                            # reciprocal-rank fusion (hybrid retrieval)
    s = Counter()
    for r in rankings:
        for rank, i in enumerate(r): s[i] += 1 / (k + rank + 1)
    return [i for i, _ in s.most_common()]
def rerank(q, cand):                                 # feature reranker: stand-in for a cross-encoder
    qt = set(terms(q))
    def score(i):
        name = set(terms(docs[i][0].replace(".", " ").replace("_", " ")))
        first = set(terms(docs[i][2].split(":", 1)[1].split(".")[0]))   # the docstring's summary sentence
        return 2.0 * len(qt & name) + 1.0 * len(qt & first) + 0.5 * bm_cache[i] / (bm_cache.max() + 1e-9)
    return sorted(cand, key=score, reverse=True)

# ---------------- 3. evaluation set: 30 questions with known answers ----------------
QA = [("get the folder my program is currently running in", ["os.getcwd"]),
      ("delete a folder and everything inside it", ["shutil.rmtree"]),
      ("copy a file to another location", ["shutil.copy", "shutil.copyfile", "shutil.copy2"]),
      ("turn a python dictionary into a JSON string", ["json.dumps"]),
      ("parse JSON text into python objects", ["json.loads"]),
      ("find every match of a regular expression in a string", ["re.findall", "re.finditer"]),
      ("replace text that matches a pattern", ["re.sub", "re.subn"]),
      ("square root of a number", ["math.sqrt", "math.isqrt"]),
      ("middle value of a list of numbers", ["statistics.median"]),
      ("pick a random element from a list", ["random.choice"]),
      ("randomly reorder a list in place", ["random.shuffle"]),
      ("pause the program for a few seconds", ["time.sleep"]),
      ("all possible orderings of some items", ["itertools.permutations"]),
      ("remember results of an expensive function call", ["functools.lru_cache", "functools.cache"]),
      ("count how many times each item appears", ["collections.Counter"]),
      ("run a shell command and capture its output", ["subprocess.run", "subprocess.check_output", "subprocess.getoutput"]),
      ("find files whose names match a wildcard like *.txt", ["glob.glob", "glob.iglob", "fnmatch.filter"]),
      ("compute a SHA-256 digest", ["hashlib.sha256"]),
      ("encode bytes as base64", ["base64.b64encode", "base64.standard_b64encode"]),
      ("wrap a long paragraph to a fixed line width", ["textwrap.wrap", "textwrap.fill"]),
      ("the n smallest items in a collection", ["heapq.nsmallest"]),
      ("insert into a sorted list and keep it sorted", ["bisect.insort", "bisect.insort_right", "bisect.insort_left"]),
      ("split a URL into scheme, host and path", ["urllib.parse.urlparse", "urllib.parse.urlsplit"]),
      ("create a temporary directory", ["tempfile.mkdtemp", "tempfile.TemporaryDirectory"]),
      ("generate a random unique identifier", ["uuid.uuid4"]),
      ("duplicate an object including everything it contains", ["copy.deepcopy"]),
      ("how similar are two strings", ["difflib.SequenceMatcher", "difflib.get_close_matches"]),
      ("check whether a path exists on disk", ["os.path.exists", "os.path.lexists"]),
      ("list the entries in a directory", ["os.listdir", "os.scandir"]),
      ("combine path components into one path", ["os.path.join"])]
ids = [d[0] for d in docs]
QA = [(q, [a for a in ans if a in ids]) for q, ans in QA]
assert all(a for _, a in QA), "every question needs at least one answer in the corpus"

def metrics(rank_fn):
    rec5 = mrr = ndcg = 0.0
    for q, ans in QA:
        r = [ids[i] for i in rank_fn(q)[:10]]
        rel = [1 if x in ans else 0 for x in r]
        rec5 += any(rel[:5]); mrr += next((1 / (k + 1) for k, v in enumerate(rel) if v), 0)
        dcg = sum(v / math.log2(k + 2) for k, v in enumerate(rel))
        ndcg += dcg / sum(1 / math.log2(k + 2) for k in range(min(len(ans), 10)))
    n = len(QA); return rec5 / n, mrr / n, ndcg / n
bm_cache = None
def run_bm25(q): return list(np.argsort(-bm25.scores(q)))
def run_dense(q): return list(np.argsort(-lsa.scores(q)))
def run_hybrid(q): return rrf(run_bm25(q)[:50], run_dense(q)[:50])
def run_hybrid_rerank(q):
    global bm_cache; bm_cache = bm25.scores(q); c = run_hybrid(q)[:30]
    return rerank(q, c) + c[30:]
print(f"\nretrieval on {len(QA)} questions      Recall@5    MRR@10   nDCG@10")
for name, fn in [("BM25 (keyword)", run_bm25), ("dense (LSA vectors)", run_dense),
                 ("hybrid (BM25 + dense, RRF)", run_hybrid), ("hybrid + rerank", run_hybrid_rerank)]:
    r, m, n = metrics(fn); print(f"  {name:<30} {r:>6.0%}     {m:>6.2f}    {n:>6.2f}")
miss = [q for q, ans in QA if not set(ids[i] for i in run_hybrid(q)[:5]) & set(ans)]
print(f"  hybrid misses ({len(miss)}): " + "; ".join(miss))

# ---------------- 4. chunking: size vs recall vs context cost ----------------
by_mod = {}
for i, (fid, mod, t) in enumerate(docs): by_mod.setdefault(mod, []).append(t)
print("\nchunking (module pages split into chunks, BM25, top 5)   Recall@5   tokens in top-5 context")
for size in (40, 120, 400, 1500):
    chunks = []
    for mod, ts in by_mod.items():
        toks = " ".join(ts).split(); step = max(1, int(size * 0.8))          # 20% overlap
        for s in range(0, len(toks), step): chunks.append(" ".join(toks[s:s + size]))
    idx = BM25(chunks); hit = 0; ctx = 0
    for q, ans in QA:
        top = list(np.argsort(-idx.scores(q))[:5])
        hit += any(any(a + ":" in chunks[i] for a in ans) for i in top); ctx += sum(len(chunks[i].split()) for i in top)
    print(f"  {size:>5}-word chunks ({len(chunks):>5} chunks)                    {hit / len(QA):>5.0%}     {ctx / len(QA):>8,.0f}")

# ---------------- 5. metadata filtering ----------------
q = "load data from a file"
print(f"\nmetadata filter - query: {q!r}")
print("  no filter:     ", [ids[i] for i in run_hybrid(q)[:4]])
print("  module == json:", [ids[i] for i in run_hybrid(q) if docs[i][1] == "json"][:4])

# ---------------- 6. query rewriting, context assembly, grounded cited answer ----------------
q = "how do I remember results of an expensive function call"
print(f"\nquery: {q!r}\n  hybrid top 3: {[ids[i] for i in run_hybrid(q)[:3]]}   <- vocabulary mismatch: the docs say 'cache', not 'remember'")
rq = "cache decorator for function results"            # written by hand here; in production an LLM rewrites the query
top = run_hybrid(rq)[:3]
print(f"  rewritten as {rq!r}\n  hybrid top 3: {[ids[i] for i in top]}")
prompt = "<system>Answer only from the documents. Cite them as [n].</system>\n<documents>\n" + "\n".join(
    f"[{n + 1}] ({ids[i]}) {docs[i][2][:150]}..." for n, i in enumerate(top)) + f"\n</documents>\n<user>{q}</user>"
print(f"\nprompt to the model ({len(prompt.split())} words):\n" + prompt)
qt = set(terms(q + " " + rq)); best = None
for n, i in enumerate(top[:1]):                      # extractive "generation": the best-supported sentence of the
    for s in re.split(r"(?<=[.!?])\s+", docs[i][2].split(":", 1)[1]):   # top-ranked document, with its citation
        sc = len(qt & set(terms(s)))
        if best is None or sc > best[0]: best = (sc, s.strip(), n + 1)
answer = f"{best[1]} [{best[2]}]"
print(f"\nanswer: {answer}\ngrounded in the cited document: {best[1] in docs[top[best[2] - 1]][2]}")

Output (Python 3.11, numpy 2.x; deterministic):

corpus: 622 documents from 30 modules
index: BM25 over 2,486 terms; dense vectors of 128 dimensions

retrieval on 30 questions      Recall@5    MRR@10   nDCG@10
  BM25 (keyword)                    70%       0.55      0.58
  dense (LSA vectors)               63%       0.38      0.47
  hybrid (BM25 + dense, RRF)        73%       0.52      0.57
  hybrid + rerank                   67%       0.45      0.49
  hybrid misses (8): get the folder my program is currently running in; delete a folder and everything inside it; find every match of a regular expression in a string; replace text that matches a pattern; all possible orderings of some items; remember results of an expensive function call; compute a SHA-256 digest; duplicate an object including everything it contains

chunking (module pages split into chunks, BM25, top 5)   Recall@5   tokens in top-5 context
     40-word chunks (  775 chunks)                      63%          197
    120-word chunks (  267 chunks)                      77%          571
    400-word chunks (   90 chunks)                      87%        1,676
   1500-word chunks (   37 chunks)                      90%        4,164

metadata filter - query: 'load data from a file'
  no filter:      ['shutil.copyfileobj', 'statistics.median', 'os.readv', 'statistics.variance']
  module == json: ['json.load', 'json.loads']

query: 'how do I remember results of an expensive function call'
  hybrid top 3: ['os.getpgid', 'os.getsid', 'functools.recursive_repr']   <- vocabulary mismatch: the docs say 'cache', not 'remember'
  rewritten as 'cache decorator for function results'
  hybrid top 3: ['functools.lru_cache', 'functools.recursive_repr', 'functools.cache']

prompt to the model (70 words):
<system>Answer only from the documents. Cite them as [n].</system>
<documents>
[1] (functools.lru_cache) functools.lru_cache: Least-recently-used cache decorator. If *maxsize* is set to None, the LRU features are disabled and the cache can grow without bo...
[2] (functools.recursive_repr) functools.recursive_repr: Decorator to make a repr function return fillvalue for a recursive call...
[3] (functools.cache) functools.cache: Simple lightweight unbounded cache. Sometimes called "memoize"....
</documents>
<user>how do I remember results of an expensive function call</user>

answer: Least-recently-used cache decorator. [1]
grounded in the cited document: True

⚠️ Notes and Corrections

On the source infographic:
  • Embedding size: it pairs "text-embedding-3-large" with "768 or 1,536 dims". OpenAI's text-embedding-3-small produces 1,536 dimensions and text-embedding-3-large 3,072 by default (both can be shortened); 768 is typical of BERT-sized open models.
  • "OpenAI rerankers": we couldn't confirm a standalone OpenAI reranking model; OpenAI's file-search tool applies its own ranking internally. BGE-reranker and Cohere Rerank are standalone rerankers.
  • Filtering order: the retrieval panel shows metadata filtering after the top-k search. Filtering after the fact can leave too few results, and for permissions it's unsafe as a design; filters are usually applied during the search.
  • The example answer and query trace are illustrative, not the output of a real system.
Checked against sources: the RAG paper (Lewis et al., 2020); BM25; reciprocal rank fusion (Cormack et al., 2009, with the constant 60); HNSW; LSA; cross-encoder reranking (Nogueira & Cho); HyDE; GraphRAG; RAGAS. All "our run" figures are real outputs of the included script; the 30 evaluation questions and the query rewrite were written by hand.

🔗 References