Context Engineering: Why Contexts Fail, and the Four Moves That Fix Them

In June 2025, Andrej Karpathy posted a one-line reframe that stuck: “+1 for ‘context engineering’ over ‘prompt engineering’.” His argument was that “prompt” makes people think of a short task description typed into a chat box, when what production LLM applications actually need is closer to operating-system design — the LLM is the CPU, its context window is RAM, and context engineering is “the delicate art and science of filling the context window with just the right information for the next step.” Cognition, the company behind the AI coding agent Devin, went further and called it “effectively the #1 job of engineers building AI agents.”

That’s a strong claim for a term that’s barely a year old, and it’s worth taking seriously enough to ask two questions precisely: what, mechanically, goes wrong when you get this wrong — and what does the actual practice of getting it right look like, in a way that isn’t just “how one vendor’s coding assistant happens to do it”? Both questions have concrete, citable answers now, from independent teams working on completely different systems who converged on the same taxonomy from different directions.

Why contexts fail: four specific, named failure modes

You could reasonably ask: modern context windows are enormous — some are a million tokens. Why not just put everything relevant in and let the model sort it out?

Because longer contexts do not reliably generate better responses. Drew Breunig’s widely-cited breakdown, “How Long Contexts Fail”, names four distinct, independently observable ways an overloaded context degrades an agent, each with its own evidence:

Context poisoning — a hallucination or error enters the context and gets repeatedly referenced as if it were true. DeepMind’s Gemini 2.5 technical report documents this directly in a Pokémon-playing agent: once its in-context “goals” summary got poisoned with wrong information about the game state, the agent would fixate on pursuing an impossible objective for a very long time before recovering, if it recovered at all.

Context distraction — the context grows so long that the model leans on it instead of on what it learned during training, and starts repeating patterns from its own history rather than reasoning fresh. The same Gemini Pokémon agent showed this past roughly 100,000 tokens: instead of synthesizing a new plan, it increasingly just repeated actions from earlier in its own transcript. The ceiling is much lower for smaller models — a Databricks study found Llama 3.1 405B’s correctness starts declining around 32k tokens, well before any hard context limit.

Context confusion — superfluous content that’s technically present but irrelevant gets used by the model anyway, degrading the response. The clearest evidence here is about tools specifically: the Berkeley Function-Calling Leaderboard, a standard tool-use benchmark, shows that every model tested performs worse once you give it more than one tool to choose from, and the benchmark’s own design deliberately includes scenarios where the correct answer is “call no function” — models still call one anyway. It gets worse as models get smaller: one paper found a quantized Llama 3.1 8B failed a benchmark outright when given all 46 available tools, but succeeded when given only 19 — well within its context window the whole time. The failure wasn’t running out of room. It was that everything you put in front of a model, it has to account for.

Context clash — new information enters the context that actively contradicts something already there. A Microsoft/Salesforce paper made this concrete by taking single, complete benchmark prompts and “sharding” them into the kind of multi-turn back-and-forth a real conversation actually looks like — the same information, just assembled gradually instead of all at once. Scores dropped by an average of 39% across the models tested; OpenAI’s o3 fell from 98.1 to 64.1 on the same task. The mechanism: models commit to a guess early, based on partial information, and that wrong guess stays in the context influencing every subsequent turn — “when LLMs take a wrong turn in a conversation, they get lost and do not recover.”

These four failure modes overlap with, but aren’t identical to, what Anthropic separately calls context rot — the narrower, architectural observation that a transformer’s self-attention lets every token attend to every other token, producing n² pairwise relationships for n tokens, so recall precision degrades measurably as a context window fills, even with no bad information in it at all. (If you want the actual mechanics of that attention computation, I wrote about it from scratch here.) Context rot is the baseline tax you pay just for length; poisoning, distraction, confusion, and clash are specific, avoidable ways a context goes wrong on top of that tax. Both point at the same underlying principle: every token you put in the context is something the model has to account for, and none of it is free.

    flowchart TD
	    A["Long / overloaded context"] --> B["Context rot<br/>(architectural: n² attention,<br/>recall degrades with length)"]
	    A --> C["Poisoning<br/>bad info repeated as fact"]
	    A --> D["Distraction<br/>over-relies on history,<br/>under-uses training"]
	    A --> E["Confusion<br/>irrelevant content<br/>still gets used"]
	    A --> F["Clash<br/>contradictory info,<br/>model can't recover"]
	    B --> G["Degraded output —<br/>not a size problem,<br/>a curation problem"]
	    C --> G
	    D --> G
	    E --> G
	    F --> G

The four moves: write, select, compress, isolate

Given that longer isn’t safer, what’s the actual discipline? The clearest general answer comes from LangChain’s “Context Engineering” post, which surveyed how a wide range of production agents — Claude Code, Cursor, Windsurf, ChatGPT, Hugging Face’s deep researcher, OpenAI’s Swarm, Anthropic’s multi-agent research system — actually handle this in practice, and found the same four moves showing up everywhere, just implemented differently. This is the framework worth internalizing, because it’s vendor-neutral by construction: every technique below is a specific instance of one of these four verbs.

Write: get context out of the live window

Writing context means saving information outside the model’s live context window so it can be pulled back in later, instead of staying resident the whole time.

The simplest version is a scratchpad — a tool call that writes to a file, or a field in a runtime state object — used to persist a plan or intermediate findings within a single task. Anthropic’s multi-agent research system does this literally: the lead agent’s first move is saving its research plan to memory, specifically because if the conversation exceeds 200,000 tokens it gets truncated, and losing the plan would be worse than the truncation itself.

Memory is the same idea stretched across sessions instead of one task. This isn’t a new idea invented by any single lab — the academic lineage runs through Reflexion (agents that reflect on a completed episode and reuse the self-generated critique on the next attempt) and Generative Agents (memories synthesized periodically from accumulated experience). It’s since become a standard product feature: ChatGPT, Cursor, and Windsurf all now auto-generate and reuse memories across sessions, independent of each other and independent of any shared standard.

Select: get the right context back in

Selecting is the inverse move: pulling previously-written or externally-stored context back into the window, for the specific step that needs it.

This is where retrieval-augmented generation lives, and it’s worth being precise about what RAG actually is rather than treating it as a vague buzzword: Lewis et al.’s original 2020 paper defines it specifically as combining a pretrained parametric sequence model with a differentiable dense-vector retriever over a non-parametric memory store (in the original paper, a Wikipedia index), fine-tuned end-to-end. Every time a model vendor announces a bigger context window, a “RAG is dead” debate reliably follows — and just as reliably turns out to be wrong, precisely because of the four failure modes above: dumping an entire corpus into context doesn’t avoid confusion and clash, it guarantees them.

The same “select the relevant subset” logic applies to tools, and the numbers here are stark. A paper on tool retrieval (nicknamed “RAG-MCP”) found that once you cross roughly 30 available tool definitions, their descriptions start overlapping enough to confuse the model, and past 100 tools the model is virtually guaranteed to pick wrong — but applying RAG to the tool descriptions themselves, retrieving only the most relevant subset per query, gave up to 3x better tool-selection accuracy. A separate paper (“Less is More”) built a small LLM-driven tool recommender that first reasons about how many tools a query actually needs, then retrieves that many — and on a small 8B model, this improved Berkeley leaderboard accuracy by 44%, while also cutting power draw 18% and latency 77%, which matters specifically if you’re running at the edge rather than on a datacenter GPU.

Selection isn’t only a retrieval-quality problem — it’s also a control problem, and it can fail in ways that feel invasive rather than just wrong. Simon Willison documented ChatGPT retrieving a previously-saved memory of his location and unexpectedly injecting it into an unrelated image-generation request. Nothing about that was a hallucination; the retrieval mechanism worked exactly as designed. It selected the wrong thing at the wrong time, which is a different failure than confusion or clash — it’s a targeting problem, not a content problem.

# The core primitive under both "select" mechanisms above: given a query
# and a pool of candidates, retrieve only what's relevant. The pool is
# documents for RAG, tool schemas for tool-loadout selection — same move.
def select(query, candidate_pool, embed, k):
    query_vec = embed(query)
    scored = [(cosine_sim(query_vec, embed(c)), c) for c in candidate_pool]
    scored.sort(reverse=True)
    return [c for _, c in scored[:k]]   # top-k only — not the whole pool

Compress: keep only what’s earning its keep

Compressing means retaining only the tokens actually required to perform the task, discarding the rest — as opposed to write/select, which decide what’s in or out of the window; compress operates on what’s already in.

Summarization is the most common form. Claude Code’s “auto-compact,” which triggers past 95% context usage, is one well-documented instance of this, but it’s not unique to one product — Cognition uses a fine-tuned model specifically for compressing context at agent-to-agent handoff boundaries, a strong enough problem in their system that it justified training a dedicated model rather than prompting a general one. The hard part is never the mechanics of summarizing; it’s knowing what’s safe to compress without losing something whose importance only becomes clear several steps later.

Trimming is the blunter, non-LLM-powered sibling of summarization — hard-coded rules (drop messages older than N turns) or a small, purpose-trained model. Provence is a good concrete example: a ~1.75GB open-weight context pruner for question-answering that, given a document and a question, strips out the irrelevant parts directly. Breunig ran it against a Wikipedia article and a specific question and it cut 95% of the document while preserving exactly the passage the question needed — a few lines of code, no LLM call required:

from transformers import AutoModel

provence = AutoModel.from_pretrained(
    "naver/provence-reranker-debertav3-v1", trust_remote_code=True
)

question = "What are my options for leaving Alameda?"
pruned = provence.process(question, long_document_text)
# ~95% of the original document removed, answer-relevant span kept

Offloading is the pattern of pushing content into an external store the model can query rather than keeping it resident. Anthropic’s documentation of their “think” tool — essentially a scratchpad the model writes to mid-task — reports up to a 54% improvement on a benchmark for specialized agents when paired with a domain-specific prompt. The mechanism generalizes well beyond that one tool: anywhere a tool call returns something large (a search result set, a file, an image), storing it externally and passing back a lightweight reference — instead of the full payload — is the same move.

Isolate: split context instead of growing one window

Isolating means splitting context across multiple, separately-scoped contexts rather than accumulating everything into one ever-growing window.

The dominant form is multi-agent architecture: a lead agent delegates focused sub-tasks to sub-agents, each with its own clean context, own tools, and own system prompt, returning only a distilled result to the parent. This pattern shows up under different names in nearly every major agent framework — OpenAI’s Swarm library was built explicitly around separation-of-concerns for this reason; Anthropic’s multi-agent research system uses it to let subagents “operate in parallel with their own context windows, exploring different aspects of the question simultaneously” before condensing to the lead agent; LangGraph supports it via supervisor and swarm libraries. Whatever the framework, the underlying claim is consistent: Anthropic reports their multi-agent system, with an Opus 4 lead coordinating Sonnet 4 sub-agents, beat a single-agent baseline by 90.2% on their internal research eval, largely because splitting work across isolated windows let the system spend far more total tokens on the problem than any single window could hold without rotting.

Hugging Face’s “CodeAgent” pattern is a different, less-discussed way to isolate context: instead of a tool-calling API where every return value round-trips through the model as a JSON blob, the agent writes code that runs in a sandbox, and large return values (an image, an audio object, a big table) get assigned to a sandboxed variable instead of being serialized back into the model’s context at all. The isolation happens in the execution environment, not between agents.

The concrete implementation differs by SDK — here’s what it looks like in the Claude Agent SDK specifically, as one example of the pattern rather than the definition of it:

from claude_agent_sdk import query, ClaudeAgentOptions, AgentDefinition

agents = {
    "research-agent": AgentDefinition(
        description="Researches a topic and writes a grounded brief.",
        prompt="You produce a sourced brief — no invented specifics.",
        tools=["WebSearch", "WebFetch", "Read", "Write"],
    ),
}

async for message in query(
    prompt="Research X and write findings to brief.md",
    options=ClaudeAgentOptions(allowed_tools=["Agent"], agents=agents),
):
    if hasattr(message, "result"):
        print(message.result)  # only this — not the sub-agent's full transcript — returns

Whether it’s this SDK’s AgentDefinition, OpenAI Swarm’s agent handoffs, or a LangGraph node with its own state schema, the shape is identical: a sub-context does its work using tens of thousands of tokens if it needs to, and only a distilled result — often on the order of 1,000–2,000 tokens — crosses back into the parent’s persistent context.

    flowchart TD
	    U["Lead agent<br/>(persistent context)"] -->|"task description only"| S1["Isolated context 1<br/>own tools, own window"]
	    U -->|"task description only"| S2["Isolated context 2<br/>own tools, own window"]
	    S1 -->|"distilled result<br/>~1-2k tokens"| U
	    S2 -->|"distilled result<br/>~1-2k tokens"| U
	    S1 -. "tens of thousands of<br/>tokens, never leaves" .-> X1[( )]
	    S2 -. "tens of thousands of<br/>tokens, never leaves" .-> X2[( )]

Where this breaks down

None of these four moves is free, and the honest accounting matters as much as the technique itself.

Isolation costs tokens, sometimes a lot of them. Anthropic reports that agents generally run about 4× the token cost of a single chat turn, and multi-agent systems specifically run about 15×. That only pays for itself when a task’s value clearly exceeds that multiplier — and it’s a poor fit for domains with fewer genuinely parallelizable sub-problems and more fine-grained, real-time dependencies between steps, which is why most coding tasks (as opposed to open-ended research) tend to isolate less well than you’d hope.

Compression is a bet about the future, made in the present. Any summarization or trimming step is deciding, right now, what won’t matter later — and getting that wrong doesn’t fail loudly. The agent continues confidently with a gap it doesn’t know it has, until the missing detail surfaces as a wrong answer several turns downstream, looking like a reasoning failure when it’s actually a curation failure upstream of the reasoning.

Selection only works as well as your retrieval does, and retrieval gets harder exactly where it matters most. Windsurf’s engineering team has been candid about this for code specifically: “indexing code ≠ context retrieval… embedding search becomes unreliable as a retrieval heuristic as the size of the codebase grows,” which is why production code-retrieval systems end up combining embedding search with grep/file search, knowledge-graph retrieval, and a reranking step — not any single technique alone.

Writing state externally only helps if something later actually selects it correctly — Willison’s ChatGPT-location example is the failure mode here: the memory was written correctly and selected successfully, and it was still wrong, because selection picked the right fact for the wrong moment.

The practical upshot

Treat these four moves as a checklist, not a menu you pick one item from. A production agent typically needs more than one: writing a scratchpad and isolating a research sub-task and compressing its output and later selecting the relevant slice of that output back in, all in the same run. What ties all four together is a single question, worth asking about every piece of context before it goes in: is this token earning its keep right now? If it’s not, the fix is one of write (push it out), select (don’t pull it in yet), compress (shrink it), or isolate (let something else hold it) — not simply “add more window and hope.”

Before optimizing any of this, instrument it. You can’t tell which of the four moves will help without first seeing where your agent’s tokens are actually going and what’s actually degrading its output — evaluation and tracing come before technique, not after. That’s true whether you’re building on Claude, GPT, Gemini, or an open-weight model; the failure modes above (context rot aside, which is architecture-specific) aren’t properties of any one vendor’s models. They’re what happens whenever an autonomous loop is left to accumulate context unchecked, and the fix has turned out to be the same four moves everywhere anyone has looked closely enough to name them.