What Agents Remember: Memory, State and the Context Window
A model remembers nothing between requests. Everything an agent "knows" during a task is text in its context window, re-sent on every call, and everything it should know next week must be stored somewhere outside the model and fetched back. On long tasks the context fills with tool results and old reasoning until the agent slows, costs more and gets confused. This part explains the kinds of agent memory, how agents trim and summarise their context, the memory products competing to be the agent's long-term store, and why prompt caching has quietly become one of the most important pieces of agent infrastructure.
1. Case file
2. Four kinds of memory
Our explainer Context Engineering and Memory covers the prompting side; Agent Memory and Planning Patterns covers design patterns. This part is about the infrastructure.
3. The context fills up
Tokens in context over a long run
Each turn adds the model's output and a tool result. Pick a strategy. Our simulation with typical sizes.
4. Editing, compaction and notes
5. Long-term memory stores
| Approach | Example | How it works |
|---|---|---|
| Extract facts | Mem0 | After a conversation, extract durable facts, de-duplicate them, and retrieve them later with combined semantic, keyword and entity scoring |
| Self-managed tiers | Letta (MemGPT) | The agent itself decides what stays in its core memory and what is moved to recall or archival storage, like an operating system paging memory |
| Knowledge graph | Zep (Graphiti) | Facts stored as a graph with time attached, so the agent knows what was true when |
| Plain files + search | Memory tools, instruction files | Markdown files the agent reads, writes and greps; transparent and easy to edit |
| Search past chats | Consumer assistants | Look up relevant earlier conversations on demand instead of keeping a profile |
Which works best is contested. Mem0 reports high LoCoMo and LongMemEval scores; Zep published a rebuttal arguing its own corrected LoCoMo result is higher; Letta reported that an agent with just file-search tools scored 74.0% on LoCoMo, above Mem0's best reported graph variant โ ๏ธ. Different harnesses and judges make the numbers hard to compare (Evaluation Layer Part 3 explains why judge choice matters). The safe reading: simple, inspectable memory is a strong baseline.
6. Prompt caching: memory for the hardware
Every agent step re-sends the whole context: system prompt, tool definitions, history. Without help, the inference server recomputes the attention keys and values (the KV cache, see ML Layer Part 7) for all of it, every time. Prompt caching keeps the computed prefix on the server so the next call only processes the new tokens. Providers bill cached reads at a large discount (Anthropic charges cache reads at 10% of the base input price; OpenAI applies caching automatically at a discount). For agents, whose prompts are mostly repeated prefix, this changes the economics.
What caching saves on one agent run
Our model: each turn re-sends the prefix; cached tokens billed at the chosen fraction.
This is why agent builders keep the start of the prompt stable (system prompt, tools, then history appended at the end) and why context editing has a cost: deleting something early in the context invalidates the cache after that point.
7. Memory risks
8. The series
Inside the Agent Layer: eight parts
9. Reading the numbers
- Primary (company-reported)
- Context editing and memory tool, +29% / +39% / โ84% tokens (Anthropic, Sep 2025, internal evals). Cache-read pricing (provider pricing pages).
- Reported / disputed โ ๏ธ
- Mem0 funding (TechCrunch via secondary). LoCoMo and LongMemEval scores from Mem0, Zep and Letta, each vendor-run.
- Our illustrations
- Context simulator, caching calculator.