Context Engineering and Memory Architecture for LLMs
A model sees only what is in its context window, and it forgets everything the moment a call ends. Context engineering is the discipline of deciding what goes into that window โ which history, which memories, which documents, which tools โ how it is compressed and ordered, and what is saved for next time. This piece covers each part, and measures five memory strategies on the same 100-turn conversation to show what each one keeps and loses.
The Big Picture
Everything an assistant could use โ system instructions, a hundred turns of conversation, stored memories, documents and files, tool definitions, tool results, the current message โ is far more than it should put in front of the model on each call. A context manager sits between all that information and the model and does five jobs: relevance selection, token budgeting, history truncation and summarization, memory retrieval, and document and tool selection. Its output is the context window for one call, plus a reserved generation budget for the model's answer.
Anthropic describes the shift this way: after years of prompt engineering โ finding the right words โ the question has become "what configuration of context is most likely to generate our model's desired behavior?" Context is a finite resource with diminishing returns, so the goal is the smallest set of high-signal tokens that does the job.
Anatomy of a Context Window
A real prompt contains many different kinds of content, competing for the same space. Typical shares in a large (about 128K-token) window, from the source infographic โ illustrative ranges, which vary enormously by product:
| Component | What it holds | Typical share |
|---|---|---|
| System instructions | Behaviour, style, rules, tool-use policy | 1Kโ5K |
| Conversation history | Recent turns, verbatim | 5Kโ50K |
| Conversation summary | Older turns, compacted | 0.5Kโ5K |
| User memories | Important facts and preferences | 0.5Kโ5K |
| Retrieved documents | RAG passages, knowledge | 1Kโ20K |
| Tool definitions | Function names and schemas | 0.5Kโ5K |
| Tool results | Search results, code output, intermediate data | 1Kโ10K |
| Current user message | The latest input, with attachments | 0.1Kโ5K |
| Generation budget | Space reserved for the output | 1Kโ8K |
The last row matters: in the major APIs the output counts against the same window as the input, so a context builder that fills the window completely leaves the model no room to answer.
Conversation History Management
A long conversation can't be kept verbatim forever. The system decides what to keep, with five common strategies โ usually combined:
The result is a compact history: <summary of turns 1โ80> + key facts + <recent turns 81โ100 verbatim> + the new message. Compaction is the same process run when a conversation gets too long: identify the key information in the older part, summarize it, keep the important facts, decisions and entities, and replace the raw history with the summary. Agentic coding tools do this automatically when a session approaches the window limit.
Working Memory vs Persistent Memory
Working memory โ inside the context window
Temporary, per request or turn. Holds the current instructions, selected history, retrieved information and tool results. Cleared or replaced on the next turn, and limited by the window size.Persistent memory โ stored outside the model
Long-term storage of user preferences, important facts, summaries of past conversations, project information and connected data. Stored in databases or vector stores and retrieved when relevant.The model's weights never change during a conversation; "memory" is always the system writing text somewhere and putting some of it back into a later context. That's why memory design is a data problem โ what to write, how to structure it, how to find it again, and how to keep it current.
Memory Write and Retrieval Pipelines
Both pipelines can go wrong in ways the next section measures: writing too little (facts never stored), retrieving too much (irrelevant or outdated items crowd the window), or storing updates as new entries so the old and new values are both retrieved.
Five Strategies, Measured
Our script builds a 100-turn conversation (4,360 tokens) about planning a product. Twelve facts are planted along the way โ the database, the budget, the deadline, the pricing and so on โ and four of them are changed later ("Update: we changed the budget is $55k"). At turn 100 we ask twelve questions and check, for each strategy, whether the current value of each fact is in the assembled context, and whether an outdated value is there too. Every strategy except the first gets the same 400-token budget.
What the numbers say:
The lesson isn't "memory stores win": ours is idealized, because our facts follow a fixed phrasing that a pattern can extract. Real extraction is done by a model and makes mistakes. The lessons are that recency alone loses early facts, generic compression loses specific facts, and retrieval without handling updates brings back stale information. Production systems combine them: structured memory for durable facts, retrieval for detail, recent turns verbatim, and timestamps or versioning so newer information wins.
Token Budgeting and Priority
Different content types compete for tokens, so the builder needs an explicit priority order. A typical one:
| Priority | Component | Example allocation (128K window) |
|---|---|---|
| 1 | Safety-critical system instructions | 2K |
| 2 | Tool definitions (if using tools) | 4K |
| 3 | Recent conversation turns | 20K |
| 4 | Current user message | 2K |
| 5 | Important memories (user, project) | 4K |
| 6 | Retrieved documents (relevant) | 40K |
| 7 | Tool results and intermediate data | 10K |
| 8 | Older history (summarized) | 4K |
| โ | Generation budget (the answer) | 42K |
Our script applies a similar order greedily to shrinking budgets. At 4,000 tokens it drops the retrieved documents first (the biggest, lowest-priority item); at 2,500 it also drops tool results and the older-history summary; at 1,500 even recent turns go, while the small, high-value memories still fit. A greedy fill like this keeps a small low-priority item when a large higher-priority one doesn't fit, which is usually what you want โ and a reason to cap each component rather than rely on order alone.
Assembling the Final Prompt
The selected pieces are formatted into one structured prompt: system instructions, then the compact history, memories, retrieved documents, tool definitions and tool results, and finally the current message. Good assembly adds separators and metadata (where a document came from, when a memory was written), truncates oversized items, enforces the token budget, and validates content and removes noise. Delimiting sections clearly โ Anthropic's documentation recommends XML-style tags such as <documents> โ helps the model keep instructions, data and history apart, which also makes it harder for text inside a document to pass itself off as an instruction.
What Happens When Context Is Poor
| Symptom | Typical cause |
|---|---|
| Forgets earlier information | Important history not included (our "recent only" run) |
| Ignores instructions | Instructions pushed out, truncated, or buried in a long context |
| Irrelevant or noisy retrieval | Weak search or no re-ranking |
| Context dilution | Too many low-quality documents or tool outputs |
| Contradictory or outdated answers | Old and new values both in context (our retrieval run) |
| Hallucination despite retrieval | Retrieved text is irrelevant or low quality |
| Wrong tool usage | Missing, unclear or overlapping tool definitions |
| Exceeds the context window | No token budget management |
Two research results explain why "just use a bigger window" doesn't fix these. Liu et al. found that models use information at the start and end of a long context more reliably than information in the middle ("Lost in the Middle"). Chroma's 2025 "Context Rot" report tested 18 models and found performance becoming increasingly unreliable as input length grows, even on simple tasks and well before the window is full. Presence isn't enough; placement and quantity matter.
Prompt and Context Caching
Repeated context prefixes can be cached so they aren't reprocessed on every call: long system prompts, tool definitions, documents that many requests share. The first request processes the full prompt and stores the processed prefix โ in practice, the attention key/value states for those tokens; later requests that begin with exactly the same tokens reuse it, cutting latency and cost (see Production LLM System Design for the OpenAI and Anthropic specifics).
The catch is in "exactly the same tokens, from the start". Our script builds 21 consecutive prompts three ways and measures how much of each one matches the previous prompt's prefix:
| Prompt order | Reusable prefix (our run) |
|---|---|
| stable-first | 76% of each prompt |
| timestamp at top | 0% of each prompt |
| retrieved docs first | 1% of each prompt |
Stable content first โ system instructions, tool definitions, memories โ then the parts that change. Put anything that changes every turn, even a one-line timestamp, at the top and nothing after it can be cached.
Context Engineering vs Prompt Engineering
Prompt engineering
Focuses on wording a single instruction or template, to improve model behaviour for a task. Important, but only one part of a bigger picture.Context engineering
Manages everything in the context window: decides what information to include, retrieve, summarize, and how to format it; balances the token budget and relevance; covers system instructions, history, memory, retrieval, tools and the current input โ across every turn, not just one.Agent Memory Architecture
Agents that work for hours or across sessions need richer memory than chat history. The CoALA framework (Sumers et al., 2023) borrows from cognitive science and splits it into a short-term working memory and long-term memories:
| Memory type | Holds | Example |
|---|---|---|
| Working (short-term) | Current task state, intermediate reasoning | The plan and progress for the task in hand |
| Episodic | Past experiences | What happened on earlier tasks, and how they turned out |
| Semantic | Knowledge and facts | Facts about the user, the codebase, the domain |
| Procedural | Skills: how to do things | Learned procedures, tool-use patterns โ and the agent's own code and prompts |
Practical systems add two stores that CoALA doesn't list as separate types: an artifact store for the documents, code and reports the agent produces, and external databases โ company data and APIs reached through tools or connectors such as the Model Context Protocol. Anthropic's context-engineering guidance for long-running agents describes the same toolkit: compaction when the window fills, structured note-taking to a file outside the context, and sub-agents that explore in their own clean context and return only a condensed result.
Run the Experiments
One script, standard library only, runs in under a second. It builds the 100-turn conversation, runs five history strategies at the same budget, applies priority budgeting at four budgets, and measures the cacheable prefix for three prompt orders.
"""Context engineering, measured: which history strategy keeps the facts a model needs, within a token budget? A simulated 100-turn conversation plants 12 facts (4 of them later changed). At turn 100 we ask 12 questions and check, for each strategy, whether the *current* value of each fact made it into the assembled context, and whether an outdated value is sitting there too. This measures what the model gets to see, not what a model would answer - but a model can't use a fact that isn't in its context. Standard library only. """ import math, random, re from collections import Counter random.seed(0) def ntok(text): return len(re.findall(r"\w+|[^\w\s]", text)) # crude token count # ---------------- a long conversation with planted (and later updated) facts ---------------- FACTS = [ # (key, turn, value, question) ("product", 3, "a scheduling SaaS for dental clinics", "What are we building?"), ("customers", 7, "small clinics with 1-5 dentists", "Who are the target customers?"), ("database", 12, "Postgres", "Which database did we choose?"), ("frontend", 18, "React", "Which frontend framework?"), ("budget", 22, "$40k", "What is the budget?"), ("deadline", 27, "March 3", "When is the deadline?"), ("style", 31, "concise, technical answers", "How does the user like answers?"), ("pricing", 38, "$49 per clinic per month", "What is the pricing?"), ("region", 44, "EU only at launch", "Where do we launch?"), ("compliance", 51, "GDPR review before launch", "What compliance step is needed?"), ("cofounder", 57, "Priya handles sales", "Who handles sales?"), ("metric", 63, "weekly active clinics", "What is the north-star metric?"), ] UPDATES = {"database": (70, "Postgres with Supabase hosting"), "budget": (76, "$55k"), "deadline": (83, "March 17"), "pricing": (90, "$59 per clinic per month")} FILLER = ["Can you explain how {x} works in more detail?", "Thanks, that makes sense.", "Let's look at {x} next.", "Draft a short plan for {x}.", "What are the risks with {x}?", "Summarize what we said about {x}.", "Here are my notes on {x}: it's going fine, nothing blocking yet.", "Good point about {x}, let's keep it."] TOPICS = ["onboarding", "the calendar view", "SMS reminders", "the booking API", "testing", "the landing page", "error handling", "analytics", "the mobile layout", "logging", "the admin panel", "caching"] turns = [] fact_at = {t: (k, v) for k, t, v, _ in FACTS} | {t: (k, v) for k, (t, v) in UPDATES.items()} for t in range(1, 101): if t in fact_at: k, v = fact_at[t] verb = "Update: we changed" if k in UPDATES and UPDATES[k][0] == t else "Also remember:" user = f"{verb} the {k} is {v}." else: user = random.choice(FILLER).format(x=random.choice(TOPICS)) reply = f"Understood. " + " ".join(random.choice(["Noted.", "Here is a detailed answer with several steps and examples.", "Let me walk through the trade-offs carefully, one at a time.", "That fits the plan."]) for _ in range(random.randint(2, 5))) turns.append((t, f"User: {user}\nAssistant: {reply}")) current = {k: v for k, _, v, _ in FACTS} | {k: v for k, (_, v) in UPDATES.items()} stale = {k: v for k, _, v, _ in FACTS if k in UPDATES} QUESTIONS = [(k, q) for k, _, _, q in FACTS] full = "\n".join(txt for _, txt in turns) print(f"conversation: {len(turns)} turns, {ntok(full):,} tokens; 12 facts planted, 4 later updated") BUDGET = 400 # tokens available for history/memory in every strategy below def recent(budget): out, used = [], 0 for _, txt in reversed(turns): n = ntok(txt) if used + n > budget: break out.insert(0, txt); used += n return out def strat_full(q): return full def strat_recent(q): return "\n".join(recent(BUDGET)) def strat_summary(q): # generic lossy summary of older turns + recent turns verbatim keep = recent(BUDGET // 2); older = [txt for _, txt in turns[:len(turns) - len(keep)]] summ = [] for txt in older: # a generic summarizer: keep the first 8 words of each user message u = txt.split("\n")[0][6:]; summ.append(" ".join(u.split()[:8])) s = "Summary of earlier turns: " + "; ".join(summ) while ntok(s) > BUDGET // 2: summ.pop(0); s = "Summary of earlier turns: " + "; ".join(summ) return s + "\n" + "\n".join(keep) MEM = {} for _, txt in turns: # memory write pipeline: extract -> structure -> store (latest value wins) m = re.search(r"(?:Also remember:|Update: we changed) the (\w+) is (.+?)\.\n", txt) if m: MEM[m.group(1)] = m.group(2) def strat_memory(q): mem = "Known facts: " + "; ".join(f"{k}: {v}" for k, v in MEM.items()) return mem + "\n" + "\n".join(recent(BUDGET - ntok(mem))) STOP = set("a an and are be do does for in is it me my of on the this to what when who which how did we where".split()) class BM25: def __init__(self, docs): self.docs = docs; self.toks = [[w for w in re.findall(r"\w+", d.lower())] for d in docs] self.avg = sum(map(len, self.toks)) / len(docs); self.df = Counter(w for t in self.toks for w in set(t)) def top(self, q, k): qw = [w for w in re.findall(r"\w+", q.lower()) if w not in STOP]; N = len(self.docs); sc = [] for i, t in enumerate(self.toks): tf = Counter(t); s = sum(math.log(1 + (N - self.df[w] + .5) / (self.df[w] + .5)) * tf[w] * 2.5 / (tf[w] + 1.5 * (.25 + .75 * len(t) / self.avg)) for w in qw if w in tf) sc.append((s, i)) return [i for s, i in sorted(sc, reverse=True)[:k] if s > 0] INDEX = BM25([txt for _, txt in turns]) def strat_retrieval(q): # retrieve the 3 most relevant older turns for this question + recent turns got = [turns[i][1] for i in sorted(INDEX.top(q, 3))] return "\n".join(got) + "\n" + "\n".join(recent(BUDGET - sum(map(ntok, got)))) KEYQ = {"product": "building product", "customers": "target customers", "database": "database", "frontend": "frontend framework", "budget": "budget", "deadline": "deadline", "style": "style answers", "pricing": "pricing", "region": "launch region", "compliance": "compliance", "cofounder": "cofounder sales", "metric": "north-star metric"} print(f"\nstrategy tokens facts present (current value) outdated value also present") for name, fn in [("full history (no budget)", strat_full), ("recent turns only", strat_recent), ("summary + recent", strat_summary), ("memory store + recent", strat_memory), ("retrieval + recent", strat_retrieval)]: toks, hit, stale_n = [], 0, 0 for k, q in QUESTIONS: ctx = fn(q + " " + KEYQ[k]); toks.append(ntok(ctx)) hit += current[k] in ctx stale_n += k in stale and (f"the {k} is {stale[k]}." in ctx) print(f" {name:<26} {sum(toks) / len(toks):>7,.0f} {hit:>2} / 12 ({hit / 12:>4.0%})" f" {stale_n} of 4 updated facts") # ---------------- priority budgeting: what survives as the budget shrinks ---------------- PARTS = [ # (priority, name, tokens) - lower number = kept first (1, "system instructions", 300), (2, "tool definitions", 500), (3, "current user message", 60), (4, "recent turns", 1200), (5, "user memories", 250), (6, "retrieved documents", 2500), (7, "tool results", 900), (8, "summary of older turns", 400)] print("\ncontext budget -> components kept (in priority order); dropped") for budget in (8000, 4000, 2500, 1500): kept, dropped, used = [], [], 0 for p, name, n in sorted(PARTS): if used + n <= budget: kept.append(name); used += n else: dropped.append(name) print(f" {budget:>5}: kept {used:>5} tokens; dropped: {', '.join(dropped) or 'nothing'}") # ---------------- prompt caching: how much of each prompt is a reusable prefix? ---------------- SYSTEM, TOOLS = "You are a helpful assistant. " * 40, "tool: search(query) tool: calc(expr) " * 30 def prompt(turn, order): hist = "\n".join(txt for _, txt in turns[max(0, turn - 6):turn - 1]) user = turns[turn - 1][1].split("\n")[0] mem = "Known facts: " + "; ".join(f"{k}: {v}" for k, v in MEM.items()) stamp = f"Current time: turn {turn}." parts = {"stable-first": [SYSTEM, TOOLS, mem, hist, user], "timestamp at top": [stamp, SYSTEM, TOOLS, mem, hist, user], "retrieved docs first": [f"Relevant: {turns[(turn * 7) % 100][1]}", SYSTEM, TOOLS, mem, hist, user]}[order] return re.findall(r"\w+|[^\w\s]", "\n".join(parts)) print("\nreusable (cacheable) prefix, averaged over turns 20-40") for order in ("stable-first", "timestamp at top", "retrieved docs first"): fr = [] for t in range(20, 41): a, b = prompt(t - 1, order), prompt(t, order) k = 0 while k < min(len(a), len(b)) and a[k] == b[k]: k += 1 fr.append(k / len(b)) print(f" {order:<22} {sum(fr) / len(fr):>5.0%} of each prompt matches the previous prompt's prefix")
Output (deterministic โ the conversation uses a fixed random seed):
conversation: 100 turns, 4,360 tokens; 12 facts planted, 4 later updated strategy tokens facts present (current value) outdated value also present full history (no budget) 4,360 12 / 12 (100%) 4 of 4 updated facts recent turns only 367 0 / 12 ( 0%) 0 of 4 updated facts summary + recent 388 1 / 12 ( 8%) 0 of 4 updated facts memory store + recent 349 12 / 12 (100%) 0 of 4 updated facts retrieval + recent 369 12 / 12 (100%) 4 of 4 updated facts context budget -> components kept (in priority order); dropped 8000: kept 6110 tokens; dropped: nothing 4000: kept 3610 tokens; dropped: retrieved documents 2500: kept 2310 tokens; dropped: retrieved documents, tool results, summary of older turns 1500: kept 1110 tokens; dropped: recent turns, retrieved documents, tool results, summary of older turns reusable (cacheable) prefix, averaged over turns 20-40 stable-first 76% of each prompt matches the previous prompt's prefix timestamp at top 0% of each prompt matches the previous prompt's prefix retrieved docs first 1% of each prompt matches the previous prompt's prefix
โ ๏ธ Notes on the Source Infographic
- "Cache prompt prefix (or KV cache)" suggests two alternatives. They're the same thing: prompt caching stores the key/value attention states computed for the prefix.
- The memory types in the "Agent Memory Architecture" panel mix a published taxonomy with practical additions: working, episodic, semantic and procedural memory come from cognitive science via the CoALA framework; "artifact memory" and "external databases" are useful engineering categories but not part of that taxonomy.
- The token-share ranges and the 128K allocation example are illustrative. Real products vary widely, and the percentage column sums to 101% because of rounding.
- A failure mode missing from its list: outdated facts retrieved alongside updated ones, which our experiment shows is the default behaviour of retrieval over raw history.