Home โ€บ Blog โ€บ Context Engineering and Memory
Deep Dive ยท With runnable experiments ๐Ÿงฉ

Context Engineering and Memory Architecture for LLMs

A model sees only what is in its context window, and it forgets everything the moment a call ends. Context engineering is the discipline of deciding what goes into that window โ€” which history, which memories, which documents, which tools โ€” how it is compressed and ordered, and what is saved for next time. This piece covers each part, and measures five memory strategies on the same 100-turn conversation to show what each one keeps and loses.

How to read this: "Our run" numbers come from the script in Run the Experiments. It measures whether the facts a model needs reach its context โ€” not what a model would answer, because no model runs in it. Token-share ranges quoted from the source infographic are illustrative, not measurements. This is the eighth piece in the How LLMs Are Built series and picks up where Production LLM System Design introduced the context builder.

The Big Picture

Everything an assistant could use โ€” system instructions, a hundred turns of conversation, stored memories, documents and files, tool definitions, tool results, the current message โ€” is far more than it should put in front of the model on each call. A context manager sits between all that information and the model and does five jobs: relevance selection, token budgeting, history truncation and summarization, memory retrieval, and document and tool selection. Its output is the context window for one call, plus a reserved generation budget for the model's answer.

Anthropic describes the shift this way: after years of prompt engineering โ€” finding the right words โ€” the question has become "what configuration of context is most likely to generate our model's desired behavior?" Context is a finite resource with diminishing returns, so the goal is the smallest set of high-signal tokens that does the job.

Anatomy of a Context Window

A real prompt contains many different kinds of content, competing for the same space. Typical shares in a large (about 128K-token) window, from the source infographic โ€” illustrative ranges, which vary enormously by product:

ComponentWhat it holdsTypical share
System instructionsBehaviour, style, rules, tool-use policy1Kโ€“5K
Conversation historyRecent turns, verbatim5Kโ€“50K
Conversation summaryOlder turns, compacted0.5Kโ€“5K
User memoriesImportant facts and preferences0.5Kโ€“5K
Retrieved documentsRAG passages, knowledge1Kโ€“20K
Tool definitionsFunction names and schemas0.5Kโ€“5K
Tool resultsSearch results, code output, intermediate data1Kโ€“10K
Current user messageThe latest input, with attachments0.1Kโ€“5K
Generation budgetSpace reserved for the output1Kโ€“8K

The last row matters: in the major APIs the output counts against the same window as the input, so a context builder that fills the window completely leaves the model no room to answer.

Conversation History Management

A long conversation can't be kept verbatim forever. The system decides what to keep, with five common strategies โ€” usually combined:

1Keep recent turns verbatim โ€” the last 5โ€“20 turns, where most of the immediate context lives.
2Summarize older turns โ€” compress them to a few hundred tokens.
3Retain important facts โ€” entities, preferences and decisions, extracted into structured memory.
4Drop irrelevant content โ€” small talk, redundant restatements, stale tool output.
5Use a memory store โ€” persist key information outside the context and bring it back when relevant.

The result is a compact history: <summary of turns 1โ€“80> + key facts + <recent turns 81โ€“100 verbatim> + the new message. Compaction is the same process run when a conversation gets too long: identify the key information in the older part, summarize it, keep the important facts, decisions and entities, and replace the raw history with the summary. Agentic coding tools do this automatically when a session approaches the window limit.

Working Memory vs Persistent Memory

Working memory โ€” inside the context window

Temporary, per request or turn. Holds the current instructions, selected history, retrieved information and tool results. Cleared or replaced on the next turn, and limited by the window size.

Persistent memory โ€” stored outside the model

Long-term storage of user preferences, important facts, summaries of past conversations, project information and connected data. Stored in databases or vector stores and retrieved when relevant.

The model's weights never change during a conversation; "memory" is always the system writing text somewhere and putting some of it back into a later context. That's why memory design is a data problem โ€” what to write, how to structure it, how to find it again, and how to keep it current.

Memory Write and Retrieval Pipelines

Write: conversation โ†’ store
Identify useful information (facts, preferences, decisions) โ†’ summarize and structure it (for example as keyโ€“value facts) โ†’ store it in a vector database or structured database. Updates should replace old values, not pile up beside them.
Retrieve: query โ†’ context
Take the current query โ†’ search for relevant memories (the same machinery as RAG) (by meaning and by metadata such as user, project, recency) โ†’ keep the top-k โ†’ insert them into the context window.

Both pipelines can go wrong in ways the next section measures: writing too little (facts never stored), retrieving too much (irrelevant or outdated items crowd the window), or storing updates as new entries so the old and new values are both retrieved.

Five Strategies, Measured

Our script builds a 100-turn conversation (4,360 tokens) about planning a product. Twelve facts are planted along the way โ€” the database, the budget, the deadline, the pricing and so on โ€” and four of them are changed later ("Update: we changed the budget is $55k"). At turn 100 we ask twelve questions and check, for each strategy, whether the current value of each fact is in the assembled context, and whether an outdated value is there too. Every strategy except the first gets the same 400-token budget.

full history (no budget) 4,360 tokens 12/12 4 outdated recent turns only 367 tokens 0/12 summary + recent 388 tokens 1/12 memory store + recent 349 tokens 12/12 retrieval + recent 369 tokens 12/12 4 outdated
Our run: how many of the 12 needed facts (current values) each strategy put into the context, at a 400-token budget. The orange count is updated facts whose outdated value was also present โ€” a contradiction the model would have to resolve.

What the numbers say:

1Full history has everything but costs 11ร— the budget, and it contains all four outdated values alongside the new ones.
2Recent turns only โ€” a sliding window โ€” found none of the 12 facts. Everything important was said earlier.
3A generic summary (keep the first words of each old turn, trimmed to fit) found one. A summarizer that doesn't know what matters keeps the wrong words, and under a tight budget it can't keep enough of them.
4A structured memory store found all 12, with no outdated values, in fewer tokens than any other strategy. Extracting facts as keyโ€“value pairs, where an update replaces the old value, is what made the difference.
5Retrieval over history also found all 12 โ€” but it retrieved the original statement and the update together for every changed fact, so the context contradicts itself four times.

The lesson isn't "memory stores win": ours is idealized, because our facts follow a fixed phrasing that a pattern can extract. Real extraction is done by a model and makes mistakes. The lessons are that recency alone loses early facts, generic compression loses specific facts, and retrieval without handling updates brings back stale information. Production systems combine them: structured memory for durable facts, retrieval for detail, recent turns verbatim, and timestamps or versioning so newer information wins.

Token Budgeting and Priority

Different content types compete for tokens, so the builder needs an explicit priority order. A typical one:

PriorityComponentExample allocation (128K window)
1Safety-critical system instructions2K
2Tool definitions (if using tools)4K
3Recent conversation turns20K
4Current user message2K
5Important memories (user, project)4K
6Retrieved documents (relevant)40K
7Tool results and intermediate data10K
8Older history (summarized)4K
โ€”Generation budget (the answer)42K

Our script applies a similar order greedily to shrinking budgets. At 4,000 tokens it drops the retrieved documents first (the biggest, lowest-priority item); at 2,500 it also drops tool results and the older-history summary; at 1,500 even recent turns go, while the small, high-value memories still fit. A greedy fill like this keeps a small low-priority item when a large higher-priority one doesn't fit, which is usually what you want โ€” and a reason to cap each component rather than rely on order alone.

Assembling the Final Prompt

The selected pieces are formatted into one structured prompt: system instructions, then the compact history, memories, retrieved documents, tool definitions and tool results, and finally the current message. Good assembly adds separators and metadata (where a document came from, when a memory was written), truncates oversized items, enforces the token budget, and validates content and removes noise. Delimiting sections clearly โ€” Anthropic's documentation recommends XML-style tags such as <documents> โ€” helps the model keep instructions, data and history apart, which also makes it harder for text inside a document to pass itself off as an instruction.

What Happens When Context Is Poor

SymptomTypical cause
Forgets earlier informationImportant history not included (our "recent only" run)
Ignores instructionsInstructions pushed out, truncated, or buried in a long context
Irrelevant or noisy retrievalWeak search or no re-ranking
Context dilutionToo many low-quality documents or tool outputs
Contradictory or outdated answersOld and new values both in context (our retrieval run)
Hallucination despite retrievalRetrieved text is irrelevant or low quality
Wrong tool usageMissing, unclear or overlapping tool definitions
Exceeds the context windowNo token budget management

Two research results explain why "just use a bigger window" doesn't fix these. Liu et al. found that models use information at the start and end of a long context more reliably than information in the middle ("Lost in the Middle"). Chroma's 2025 "Context Rot" report tested 18 models and found performance becoming increasingly unreliable as input length grows, even on simple tasks and well before the window is full. Presence isn't enough; placement and quantity matter.

Prompt and Context Caching

Repeated context prefixes can be cached so they aren't reprocessed on every call: long system prompts, tool definitions, documents that many requests share. The first request processes the full prompt and stores the processed prefix โ€” in practice, the attention key/value states for those tokens; later requests that begin with exactly the same tokens reuse it, cutting latency and cost (see Production LLM System Design for the OpenAI and Anthropic specifics).

The catch is in "exactly the same tokens, from the start". Our script builds 21 consecutive prompts three ways and measures how much of each one matches the previous prompt's prefix:

Prompt orderReusable prefix (our run)
stable-first76% of each prompt
timestamp at top0% of each prompt
retrieved docs first1% of each prompt

Stable content first โ€” system instructions, tool definitions, memories โ€” then the parts that change. Put anything that changes every turn, even a one-line timestamp, at the top and nothing after it can be cached.

Context Engineering vs Prompt Engineering

Prompt engineering

Focuses on wording a single instruction or template, to improve model behaviour for a task. Important, but only one part of a bigger picture.

Context engineering

Manages everything in the context window: decides what information to include, retrieve, summarize, and how to format it; balances the token budget and relevance; covers system instructions, history, memory, retrieval, tools and the current input โ€” across every turn, not just one.

Agent Memory Architecture

Agents that work for hours or across sessions need richer memory than chat history. The CoALA framework (Sumers et al., 2023) borrows from cognitive science and splits it into a short-term working memory and long-term memories:

Memory typeHoldsExample
Working (short-term)Current task state, intermediate reasoningThe plan and progress for the task in hand
EpisodicPast experiencesWhat happened on earlier tasks, and how they turned out
SemanticKnowledge and factsFacts about the user, the codebase, the domain
ProceduralSkills: how to do thingsLearned procedures, tool-use patterns โ€” and the agent's own code and prompts

Practical systems add two stores that CoALA doesn't list as separate types: an artifact store for the documents, code and reports the agent produces, and external databases โ€” company data and APIs reached through tools or connectors such as the Model Context Protocol. Anthropic's context-engineering guidance for long-running agents describes the same toolkit: compaction when the window fills, structured note-taking to a file outside the context, and sub-agents that explore in their own clean context and return only a condensed result.

Context engineering in one paragraph
The model only knows what's in its window, so every call is assembled: stable instructions and tool definitions first (so they can be cached), then durable facts from a structured memory store, relevant retrieved passages, a summary of older history, recent turns verbatim and the new message โ€” each capped by a token budget with an explicit priority order, with room left for the answer. Facts should be extracted and updated in place rather than left to recency or a generic summary, retrieval needs a way for newer information to win, and more context is not automatically better: models use long contexts unevenly, so the smallest high-signal context is the goal.

Run the Experiments

One script, standard library only, runs in under a second. It builds the 100-turn conversation, runs five history strategies at the same budget, applies priority budgeting at four budgets, and measures the cacheable prefix for three prompt orders.

"""Context engineering, measured: which history strategy keeps the facts a model needs, within a token budget?

A simulated 100-turn conversation plants 12 facts (4 of them later changed). At turn 100 we ask 12 questions
and check, for each strategy, whether the *current* value of each fact made it into the assembled context,
and whether an outdated value is sitting there too. This measures what the model gets to see, not what a
model would answer - but a model can't use a fact that isn't in its context.
Standard library only.
"""
import math, random, re
from collections import Counter

random.seed(0)
def ntok(text): return len(re.findall(r"\w+|[^\w\s]", text))      # crude token count

# ---------------- a long conversation with planted (and later updated) facts ----------------
FACTS = [  # (key, turn, value, question)
    ("product", 3, "a scheduling SaaS for dental clinics", "What are we building?"),
    ("customers", 7, "small clinics with 1-5 dentists", "Who are the target customers?"),
    ("database", 12, "Postgres", "Which database did we choose?"),
    ("frontend", 18, "React", "Which frontend framework?"),
    ("budget", 22, "$40k", "What is the budget?"),
    ("deadline", 27, "March 3", "When is the deadline?"),
    ("style", 31, "concise, technical answers", "How does the user like answers?"),
    ("pricing", 38, "$49 per clinic per month", "What is the pricing?"),
    ("region", 44, "EU only at launch", "Where do we launch?"),
    ("compliance", 51, "GDPR review before launch", "What compliance step is needed?"),
    ("cofounder", 57, "Priya handles sales", "Who handles sales?"),
    ("metric", 63, "weekly active clinics", "What is the north-star metric?"),
]
UPDATES = {"database": (70, "Postgres with Supabase hosting"), "budget": (76, "$55k"),
           "deadline": (83, "March 17"), "pricing": (90, "$59 per clinic per month")}
FILLER = ["Can you explain how {x} works in more detail?", "Thanks, that makes sense.", "Let's look at {x} next.",
          "Draft a short plan for {x}.", "What are the risks with {x}?", "Summarize what we said about {x}.",
          "Here are my notes on {x}: it's going fine, nothing blocking yet.", "Good point about {x}, let's keep it."]
TOPICS = ["onboarding", "the calendar view", "SMS reminders", "the booking API", "testing", "the landing page",
          "error handling", "analytics", "the mobile layout", "logging", "the admin panel", "caching"]
turns = []
fact_at = {t: (k, v) for k, t, v, _ in FACTS} | {t: (k, v) for k, (t, v) in UPDATES.items()}
for t in range(1, 101):
    if t in fact_at:
        k, v = fact_at[t]
        verb = "Update: we changed" if k in UPDATES and UPDATES[k][0] == t else "Also remember:"
        user = f"{verb} the {k} is {v}."
    else:
        user = random.choice(FILLER).format(x=random.choice(TOPICS))
    reply = f"Understood. " + " ".join(random.choice(["Noted.", "Here is a detailed answer with several steps and examples.",
            "Let me walk through the trade-offs carefully, one at a time.", "That fits the plan."]) for _ in range(random.randint(2, 5)))
    turns.append((t, f"User: {user}\nAssistant: {reply}"))
current = {k: v for k, _, v, _ in FACTS} | {k: v for k, (_, v) in UPDATES.items()}
stale = {k: v for k, _, v, _ in FACTS if k in UPDATES}
QUESTIONS = [(k, q) for k, _, _, q in FACTS]
full = "\n".join(txt for _, txt in turns)
print(f"conversation: {len(turns)} turns, {ntok(full):,} tokens; 12 facts planted, 4 later updated")

BUDGET = 400    # tokens available for history/memory in every strategy below

def recent(budget):
    out, used = [], 0
    for _, txt in reversed(turns):
        n = ntok(txt)
        if used + n > budget: break
        out.insert(0, txt); used += n
    return out

def strat_full(q):                    return full
def strat_recent(q):                  return "\n".join(recent(BUDGET))
def strat_summary(q):                 # generic lossy summary of older turns + recent turns verbatim
    keep = recent(BUDGET // 2); older = [txt for _, txt in turns[:len(turns) - len(keep)]]
    summ = []
    for txt in older:                 # a generic summarizer: keep the first 8 words of each user message
        u = txt.split("\n")[0][6:]; summ.append(" ".join(u.split()[:8]))
    s = "Summary of earlier turns: " + "; ".join(summ)
    while ntok(s) > BUDGET // 2: summ.pop(0); s = "Summary of earlier turns: " + "; ".join(summ)
    return s + "\n" + "\n".join(keep)
MEM = {}
for _, txt in turns:                  # memory write pipeline: extract -> structure -> store (latest value wins)
    m = re.search(r"(?:Also remember:|Update: we changed) the (\w+) is (.+?)\.\n", txt)
    if m: MEM[m.group(1)] = m.group(2)
def strat_memory(q):
    mem = "Known facts: " + "; ".join(f"{k}: {v}" for k, v in MEM.items())
    return mem + "\n" + "\n".join(recent(BUDGET - ntok(mem)))
STOP = set("a an and are be do does for in is it me my of on the this to what when who which how did we where".split())
class BM25:
    def __init__(self, docs):
        self.docs = docs; self.toks = [[w for w in re.findall(r"\w+", d.lower())] for d in docs]
        self.avg = sum(map(len, self.toks)) / len(docs); self.df = Counter(w for t in self.toks for w in set(t))
    def top(self, q, k):
        qw = [w for w in re.findall(r"\w+", q.lower()) if w not in STOP]; N = len(self.docs); sc = []
        for i, t in enumerate(self.toks):
            tf = Counter(t); s = sum(math.log(1 + (N - self.df[w] + .5) / (self.df[w] + .5)) * tf[w] * 2.5 /
                                     (tf[w] + 1.5 * (.25 + .75 * len(t) / self.avg)) for w in qw if w in tf)
            sc.append((s, i))
        return [i for s, i in sorted(sc, reverse=True)[:k] if s > 0]
INDEX = BM25([txt for _, txt in turns])
def strat_retrieval(q):               # retrieve the 3 most relevant older turns for this question + recent turns
    got = [turns[i][1] for i in sorted(INDEX.top(q, 3))]
    return "\n".join(got) + "\n" + "\n".join(recent(BUDGET - sum(map(ntok, got))))

KEYQ = {"product": "building product", "customers": "target customers", "database": "database", "frontend": "frontend framework",
        "budget": "budget", "deadline": "deadline", "style": "style answers", "pricing": "pricing", "region": "launch region",
        "compliance": "compliance", "cofounder": "cofounder sales", "metric": "north-star metric"}
print(f"\nstrategy                     tokens   facts present (current value)   outdated value also present")
for name, fn in [("full history (no budget)", strat_full), ("recent turns only", strat_recent),
                 ("summary + recent", strat_summary), ("memory store + recent", strat_memory),
                 ("retrieval + recent", strat_retrieval)]:
    toks, hit, stale_n = [], 0, 0
    for k, q in QUESTIONS:
        ctx = fn(q + " " + KEYQ[k]); toks.append(ntok(ctx))
        hit += current[k] in ctx
        stale_n += k in stale and (f"the {k} is {stale[k]}." in ctx)
    print(f"  {name:<26} {sum(toks) / len(toks):>7,.0f}   {hit:>2} / 12 ({hit / 12:>4.0%})"
          f"                   {stale_n} of 4 updated facts")

# ---------------- priority budgeting: what survives as the budget shrinks ----------------
PARTS = [  # (priority, name, tokens) - lower number = kept first
    (1, "system instructions", 300), (2, "tool definitions", 500), (3, "current user message", 60),
    (4, "recent turns", 1200), (5, "user memories", 250), (6, "retrieved documents", 2500),
    (7, "tool results", 900), (8, "summary of older turns", 400)]
print("\ncontext budget -> components kept (in priority order); dropped")
for budget in (8000, 4000, 2500, 1500):
    kept, dropped, used = [], [], 0
    for p, name, n in sorted(PARTS):
        if used + n <= budget: kept.append(name); used += n
        else: dropped.append(name)
    print(f"  {budget:>5}: kept {used:>5} tokens; dropped: {', '.join(dropped) or 'nothing'}")

# ---------------- prompt caching: how much of each prompt is a reusable prefix? ----------------
SYSTEM, TOOLS = "You are a helpful assistant. " * 40, "tool: search(query) tool: calc(expr) " * 30
def prompt(turn, order):
    hist = "\n".join(txt for _, txt in turns[max(0, turn - 6):turn - 1])
    user = turns[turn - 1][1].split("\n")[0]
    mem = "Known facts: " + "; ".join(f"{k}: {v}" for k, v in MEM.items())
    stamp = f"Current time: turn {turn}."
    parts = {"stable-first": [SYSTEM, TOOLS, mem, hist, user],
             "timestamp at top": [stamp, SYSTEM, TOOLS, mem, hist, user],
             "retrieved docs first": [f"Relevant: {turns[(turn * 7) % 100][1]}", SYSTEM, TOOLS, mem, hist, user]}[order]
    return re.findall(r"\w+|[^\w\s]", "\n".join(parts))
print("\nreusable (cacheable) prefix, averaged over turns 20-40")
for order in ("stable-first", "timestamp at top", "retrieved docs first"):
    fr = []
    for t in range(20, 41):
        a, b = prompt(t - 1, order), prompt(t, order)
        k = 0
        while k < min(len(a), len(b)) and a[k] == b[k]: k += 1
        fr.append(k / len(b))
    print(f"  {order:<22} {sum(fr) / len(fr):>5.0%} of each prompt matches the previous prompt's prefix")

Output (deterministic โ€” the conversation uses a fixed random seed):

conversation: 100 turns, 4,360 tokens; 12 facts planted, 4 later updated

strategy                     tokens   facts present (current value)   outdated value also present
  full history (no budget)     4,360   12 / 12 (100%)                   4 of 4 updated facts
  recent turns only              367    0 / 12 (  0%)                   0 of 4 updated facts
  summary + recent               388    1 / 12 (  8%)                   0 of 4 updated facts
  memory store + recent          349   12 / 12 (100%)                   0 of 4 updated facts
  retrieval + recent             369   12 / 12 (100%)                   4 of 4 updated facts

context budget -> components kept (in priority order); dropped
   8000: kept  6110 tokens; dropped: nothing
   4000: kept  3610 tokens; dropped: retrieved documents
   2500: kept  2310 tokens; dropped: retrieved documents, tool results, summary of older turns
   1500: kept  1110 tokens; dropped: recent turns, retrieved documents, tool results, summary of older turns

reusable (cacheable) prefix, averaged over turns 20-40
  stable-first             76% of each prompt matches the previous prompt's prefix
  timestamp at top          0% of each prompt matches the previous prompt's prefix
  retrieved docs first      1% of each prompt matches the previous prompt's prefix

โš ๏ธ Notes on the Source Infographic

The infographic is broadly accurate; a few clarifications:
  • "Cache prompt prefix (or KV cache)" suggests two alternatives. They're the same thing: prompt caching stores the key/value attention states computed for the prefix.
  • The memory types in the "Agent Memory Architecture" panel mix a published taxonomy with practical additions: working, episodic, semantic and procedural memory come from cognitive science via the CoALA framework; "artifact memory" and "external databases" are useful engineering categories but not part of that taxonomy.
  • The token-share ranges and the 128K allocation example are illustrative. Real products vary widely, and the percentage column sums to 101% because of rounding.
  • A failure mode missing from its list: outdated facts retrieved alongside updated ones, which our experiment shows is the default behaviour of retrieval over raw history.
Checked against sources: Anthropic's "Effective context engineering for AI agents" (the prompt-vs-context framing, compaction, note-taking, sub-agents); Chroma's "Context Rot" report (July 2025, 18 models); Liu et al., "Lost in the Middle"; the CoALA memory taxonomy; Anthropic's prompt-engineering guidance on XML tags. All "our run" figures are real outputs of the included script.

๐Ÿ”— References