Codex, Taken Apart
OpenAI's Codex dismantled into eight layers: an open-source local agent and a cloud agent that runs every task in its own container and hands back a diff. One task traced through both, the design choices, failure modes, and what building one takes.
1. Case file
- In one sentence.
- Two harnesses under one name: a local agent you can read the code of, and a cloud agent you delegate tasks to that works in isolated containers and hands back diffs.
2. The whole system in one picture
๐๏ธ The whole system, layer by layer, with one task traced
Same eight-layer reference stack in every article of the series. Components and calls are drawn from the project's documentation and code; where internals are not public the drawing is inferred from documented behaviour. The monospace box shows what actually moves between layers at each step (simplified).
3. Layer by layer
L1๐ฅ๏ธ Interfaces ยท Where people (or other agents) talk to it
- What it does
- Ways in: terminal, editor, and a web app for delegating tasks.
- How it is built here
- An open-source CLI (Apache-2.0) and Codex Web at chatgpt.com/codex, plus IDE integration โ ๏ธ; sign in with a ChatGPT plan or API key.
- Design choice
- Two products sharing a name and a model: a local agent and a cloud agent.
- Without it
- Local-only means one task at a time on your laptop; cloud-only means no access to your local setup.
L2๐งญ Sessions & orchestration ยท Who owns a task: sessions, background tasks, subagents
- What it does
- Ownership of work: interactive sessions locally, delegated tasks in the cloud.
- How it is built here
- Cloud tasks run independently and in parallel; results come back for review.
- Design choice
- Delegation is the main mode in the cloud: hand off, keep working, review later.
- Without it
- Long tasks block the developer; parallel work is impossible.
L3๐ Agent loop ยท How it decides the next step
- What it does
- Plan, act, observe, repeat, per task.
- How it is built here
- An agent loop calling shell and edit tools; one loop per cloud task.
- Design choice
- Many small independent loops rather than one long conversation.
- Without it
- Tasks interfere with each other through shared state.
๐ How this loop compares with the other four, dial by dial, and its RSI level: Inside the Agent Loop
L4๐ง Context engine ยท What the model sees on each turn
- What it does
- What the model knows about your project.
- How it is built here
- AGENTS.md instructions โ ๏ธ and the repository itself, read through tools.
- Design choice
- A plain markdown convention any harness can read, instead of a proprietary format.
- Without it
- The agent guesses how to install and test, and fails in the container.
L5๐ ๏ธ Tools ยท What it can do
- What it does
- Actions: run commands and change files.
- How it is built here
- Shell commands and patch-based edits; MCP for outside tools โ ๏ธ.
- Design choice
- The shell as the universal tool.
- Without it
- Every build system needs a dedicated integration.
L6๐ก๏ธ Policy & safety ยท What it may do
- What it does
- What may happen without asking.
- How it is built here
- Approval modes from read-only to full access locally โ ๏ธ; per-task network policy in the cloud โ ๏ธ.
- Design choice
- Policy is tied to the environment: strict where the agent runs on your machine, isolated where it runs in the cloud.
- Without it
- A local agent with full access can damage your machine; a cloud agent with open network can leak code.
L7๐ฆ Execution ยท Where actions actually run
- What it does
- Where actions run.
- How it is built here
- Locally inside operating-system sandbox restrictions โ ๏ธ; in the cloud in a container per task.
- Design choice
- Isolation everywhere is the defining choice: the agent never shares an environment with another task.
- Without it
- One bad command affects everything else running.
L8๐งฌ Model & telemetry ยท Which model, and what gets logged
- What it does
- The model, and the record of what happened.
- How it is built here
- OpenAI models; logs and diffs for every cloud task.
- Design choice
- Every task ends in reviewable evidence (log + diff), not just a message.
- Without it
- You merge changes you cannot audit.
4. Connecting the dots
- Isolation is the organising idea.
- Layers 6 and 7 decide the product: a container per task makes parallel delegation safe; OS sandboxing makes the local agent tolerable.
- Evidence over conversation.
- Each task ends as a diff plus a log, which fits how teams already review code.
- Open where it is local.
- The CLI is open source, so the local half of the architecture can be read in the repository; the cloud half is documented behaviour only.
๐งฌ Design DNA: the three decisions that define it
๐ธ Where the tokens go in the traced task (our estimate)
An illustrative split for a ~300k-token task: about $1.20 at an assumed blended $4 per million tokens. Real numbers vary widely with the repo, the model and caching; the shape (context re-sent every turn dominates) is the point.
5. How it evolved
- โ ๏ธ Earlier dates are from OpenAI announcements as we recall them; OpenAI's pages could not be reached from our build environment.
6. Where it breaks, and what it does about it
| Failure | Layer | What happens | What this system does about it |
|---|---|---|---|
| Environment mismatch | L7 | The container cannot install or build the project. | AGENTS.md and setup scripts describe how to build โ ๏ธ. |
| Network needed | L6 | A task needs a package or an API the policy blocks. | Adjust the task's network policy deliberately โ ๏ธ. |
| Stale base | L2 | A long task finishes against an old branch. | Review the diff; rebase before merging. |
| Over-broad changes | L5 | Patches touch more than needed. | Conventions in AGENTS.md; review before PR. |
| Prompt injection | L5 | Repository content tries to instruct the agent. | Isolation limits what a hijacked task can reach. |
| Too much access locally | L6 | Full-access mode on a developer laptop. | Prefer the default sandboxed modes โ ๏ธ. |
7. Security and threat model
๐งฑ Trust boundaries
8. Standards it shares, and benchmarks
- Interoperability.
- AGENTS.md: a plain markdown file for agent instructions that several harnesses read.
- MCP for outside tools โ ๏ธ.
- Benchmarks.
- Agentic-coding scores combine model and harness; see the Frontier Benchmark Tracker.
9. What people use it for
10. How to build one from scratch
- Start small: isolation and diffs.
- This toy captures the Codex ideas: a fresh copy of the repo per task, AGENTS.md, an approval mode and a diff for review. Separate from the other articles' toys; the model is a scripted stub.
"""mini_codex.py โ a toy harness in the style of Codex (illustrative).
Delegation and isolation: each task gets its own copy of the repository (standing
in for a cloud container), reads AGENTS.md, works under an approval mode, and
hands back a diff for review. Several tasks run independently. The "model" is a
scripted stub; standard library only.
"""
import copy, difflib
REPO = {"AGENTS.md": "test: python test_dates.py\nconvention: minimal diffs\n",
"dates.py": "def parse(s):\n d, m, y = s.split('/')\n return (int(y), int(m), int(d))\n",
"README.md": "Date utils\n"}
MODES = {"read-only": set(), "auto": {"edit", "test"}, "full": {"edit", "test", "network"}}
PLANS = {"fix-date-test": [("test", None), ("edit", ("dates.py", " d, m, y = s.split('/')\n", " y, m, d = s.split('-')\n")), ("test", None)],
"update-readme": [("edit", ("README.md", "Date utils\n", "Date utils (ISO 8601 input)\n"))],
"fetch-deps": [("network", "pip download requests")]}
def run_task(name, mode):
box = copy.deepcopy(REPO) # isolated container per task
log = [f"read AGENTS.md: {box['AGENTS.md'].splitlines()[0]}"]
for act, arg in PLANS[name]:
if act not in MODES[mode]: log.append(f"{act}: blocked by '{mode}' mode"); continue
if act == "edit": f, old, new = arg; box[f] = box[f].replace(old, new); log.append(f"edit {f}")
elif act == "test": ok = "y, m, d" in box["dates.py"]; log.append("test: " + ("pass" if ok else "fail"))
diff = [l for f in REPO for l in difflib.unified_diff(REPO[f].splitlines(), box[f].splitlines(), f"a/{f}", f"b/{f}", lineterm="", n=0)]
return log, diff
def main():
for name, mode in (("fix-date-test", "auto"), ("update-readme", "auto"), ("fetch-deps", "auto")):
log, diff = run_task(name, mode)
print(f"task {name} [{mode}]: " + "; ".join(log))
print(" diff for review: " + (" | ".join(l for l in diff if l[:1] in "+-" and l[:3] not in ("+++", "---")) or "(no changes)"))
print(f"original repo untouched: {'d, m, y' in REPO['dates.py']}")
if __name__ == "__main__":
main()| Phase | What you build | Time* | Team* | Cost* |
|---|---|---|---|---|
| 0 ยท Prototype | Local loop with shell and patch tools | 1โ2 weeks | 1 | $15โ40k |
| 1 ยท Safe CLI | Approval modes, OS sandboxing, AGENTS.md | 2โ3 months | 2โ4 | $200โ500k |
| 2 ยท Cloud tasks | Container per task, repo cloning, setup scripts, logs | 3โ6 months | 5โ10 | $1โ2.5M |
| 3 ยท Parallel + review | Task queue, diff review UI, PR creation, network policies | 3โ4 months | 6โ12 | $1โ2M |
| 4 ยท Product | Accounts, billing, IDE, scaling to many containers, security review | 6โ12 months | 20โ50+ | $5โ15M+ a year (incl. compute) |
*Our estimates, not vendor figures: fully loaded engineer cost assumed at $250k a year (US-style salaries; lower elsewhere), plus model API and infrastructure. Treat as orders of magnitude.
- The hard part.
- Container orchestration at scale (start-up time, caching dependencies) and reproducing each customer's build environment.
- Build or extend?
- The CLI is open source: study or fork it before writing your own local agent.
๐งฎ What would running your own cost a month?
Our model: model cost = tasks ร tokens ร price ร 30 days; people = engineers ร $250k รท 12; infrastructure (sandboxes, storage, logs) assumed at 15% of model cost. A blended price mixes cached input, fresh input and output; check your provider's current prices.
11. When to choose it
โ Choose it when
- Many well-defined tasks to delegate in parallel.
- You want an open-source local agent.
- You prefer reviewing diffs to chatting.
โ ๏ธ Think twice when
- Tasks need your local environment and secrets: cloud containers will not have them.
- You want an agent that learns across sessions.
12. How it compares, layer by layer
| Harness | L1Interfaces | L2Sessions & orchestration | L3Agent loop | L4Context engine | L5Tools | L6Policy & safety | L7Execution | L8Model & telemetry |
|---|---|---|---|---|---|---|---|---|
| Claude Code | Terminal, IDE, desktop, web, Slack; Agent SDK | Sessions with resume; subagents; cloud sessions on the web | One general think โ act โ observe loop | CLAUDE.md, auto memory, compaction | Built-in file/shell/search tools, MCP, skills | Permission rules and modes, hooks | Your machine (optional sandboxing โ ๏ธ); cloud for web sessions | Claude models; transcripts, cost and usage telemetry |
| Codex | Terminal, IDE, web (chatgpt.com/codex) | Parallel cloud tasks; local sessions | Agent loop per task | AGENTS.md, repo context | Shell, file edits, MCP | Approval modes โ ๏ธ | OS sandbox locally โ ๏ธ; container per cloud task | OpenAI models; task logs and diffs |
| Gemini CLI / Jules | Terminal (CLI); browser + GitHub (Jules) | Interactive session vs asynchronous task | Agent loop (CLI); plan-then-execute (Jules โ ๏ธ) | GEMINI.md context files | Built-in tools, MCP | Confirmations in the CLI; plan approval in Jules โ ๏ธ | Your machine (CLI); cloud VM (Jules โ ๏ธ) | Gemini models (CLI "not optimized for Gemini 4+") |
| Deep Agents | Your own app (library) | LangGraph threads; sub-agents | Planning loop on LangGraph | Filesystem as memory; offloading | Your tools, MCP, skills | Human-in-the-loop interrupts | Pluggable backends: local, sandboxed, remote | Any chat model; LangGraph state and tracing |
| Hermes Agent | Terminal + Telegram, Discord, Slack, WhatsApp, Signal, email | Gateway, subagents, cron scheduler | Agent loop (details on its docs site โ ๏ธ) | Curated memory, FTS5 session search, user model | Tools, self-written skills (agentskills.io), Python RPC scripts | Depends on backend and configuration โ ๏ธ | Six backends: local, Docker, SSH, Modal, Daytona, Vercel Sandbox | Model you configure โ ๏ธ |
One line per layer; our reading of each project's documentation and code as of 7 Nov 2026.
| Harness | Runs where | Open source | Subagents | Persistent memory | Skills | MCP | Sandbox | Human approval | Async / cloud tasks |
|---|---|---|---|---|---|---|---|---|---|
| Claude Code | Terminal, IDE, desktop, web | โ | โ | โ | โ | โ | โ | โ | โ |
| Codex | Terminal, IDE, cloud | โ (CLI) | โ | โ | โ | โ | โ | โ | โ |
| Gemini CLI / Jules | Terminal (CLI), cloud VM (Jules) | โ (CLI) | โ | โ | โ | โ | โ | โ | โ |
| Deep Agents | Your Python / JS app | โ | โ | โ | โ | โ | โ | โ | โ |
| Hermes Agent | Terminal + chat apps | โ | โ | โ | โ | โ | โ | โ | โ |
โ built in ยท โ partial or via add-ons ยท โ not a feature. Our reading of each project's documentation as of 7 Nov 2026; check before relying on it.
13. The series
Agentic AI System Designs: four kinds of agent system, one teardown each
All of them on one page: Agentic Systems at a Glance.
๐ Glossary: 15 terms used across the series
- Agent loop
- The repeat cycle: the model picks an action, the harness runs it, the result goes back to the model.
- Harness
- Everything around the model that turns it into a working agent: loop, tools, context, policy, execution.
- Context window
- The text the model can see at once. Everything the agent knows on a turn has to fit in it.
- Context compaction
- Summarising older conversation so a long task keeps fitting in the context window.
- Tool schema
- A machine-readable description of a tool (name, inputs) that tells the model how to call it.
- MCP
- Model Context Protocol: an open standard for connecting agents to outside tools and data.
- Subagent
- A second agent with its own context, used for a side task; only its summary comes back.
- Skill
- A packaged set of instructions (and sometimes scripts) the agent loads when relevant.
- Hook
- Your own command that runs automatically before or after an agent action.
- Sandbox
- A restricted place to run commands so mistakes or attacks cannot reach the rest of the system.
- Approval mode
- The rule for what the agent may do without asking you first.
- Prompt injection
- Instructions hidden in data the agent reads (a file, a web page) that try to hijack it.
- Trace
- The full record of a run: every model call, tool call and result.
- AGENTS.md / CLAUDE.md / GEMINI.md
- Plain markdown files of project instructions that harnesses read at the start of a session.
- Asynchronous agent
- An agent you hand a task to and leave; it returns a result (often a pull request) later.