Agent Harnesses: Why the Same Model Scores 34 Points Apart on the Same Benchmark

Take Claude Sonnet 4.5 — same weights, same API, same day — and run it on SWE-bench Verified under two different scaffolds. Under a SWE-Agent-style harness it resolves 68% of the tasks. Under a “HAL Generalist” harness it resolves 34%. Same model. Thirty-four points. That is not a rounding error or a flaky eval — it is the single most important fact about agent benchmarks that leaderboards routinely fail to disclose: the harness is not a thin wrapper around the model, it is roughly as large a factor in the reported score as the model itself, sometimes larger.

This post is about what a “harness” (also called a scaffold, an agent-computer interface, or an agentic framework) actually is, what it owns, what a harness engineer does with their day that’s different from what a model researcher does, and why swapping it out — holding the model completely fixed — can move a benchmark score more than swapping the model itself. The last post in this arc covered how a served model actually gets more tokens out the door — batching, memory, throughput. This one is about the layer built on top of that serving layer: once you can call a model cheaply and reliably, what wraps around it to turn a stateless next-token predictor into something that reads a codebase, edits files, runs tests, and fixes its own mistakes.

The model doesn’t do anything. The harness does.

It’s worth being pedantic about this because the pedantry is the whole point. An LLM API call is a pure function: context window in, probability distribution over the next token out. It has no memory between calls, no ability to read a file, no concept of “the terminal,” and no way to know whether its last edit compiled. Every single one of those things — memory, file access, a terminal, feedback on whether an edit worked — is supplied by software sitting outside the model. That software is the harness.

Concretely, a harness owns:

• The agentic loop. Not one call — a repeated cycle: feed the model a system prompt, tool definitions, and running history; let it emit either a text answer or a tool call; execute that tool call against the real environment; feed the result back in; repeat until the model stops requesting tools or a limit is hit. Anthropic’s own Claude Code documentation describes this as “gather context, take action, verify results,” looping until no more tool calls are requested.
• The Agent-Computer Interface (ACI). A term coined by the SWE-agent paper (Yang et al., Princeton, NeurIPS 2024) for the specific surface a model is given to act through — not “hand it a raw Linux shell” but purpose-built commands: a bounded edit command instead of raw sed, a search command that returns a summarized result set instead of a raw grep dump, a file viewer windowed at a tuned size instead of the whole file. Their central, empirically validated claim: the interface the model acts through is a first-class design variable with a large, measurable effect on task success, independent of the model’s own reasoning ability.
• Tool/function-calling definitions. JSON-schema’d affordances — Read, Edit, Bash, Grep, spawn-a-subagent — injected into every request as part of the context budget. Every tool definition costs tokens on every single turn, which makes “how many tools to expose” a real design constraint, not a free lunch.
• Context and memory management. What survives across turns, what gets summarized when the window fills, and what’s persisted outside the window entirely. Claude Code re-injects project memory files on every request and prompt-caches them; when the conversation nears the context limit, the harness triggers automatic compaction — summarizing older turns while preserving recent exchanges.
• Sandboxing. Where tool calls actually execute — a container, a VM, the bare host — and what blast radius a bad command can have. OpenAI’s Codex CLI runs shell commands in a containerized sandbox by default, requiring explicit approval to touch anything outside it.
• Permission systems. Whether a tool call runs automatically, is blocked, or pauses for a human. Claude Code’s SDK exposes this as explicit permission modes (default, accept-edits, plan-only, bypass) plus a hook system that can intercept or veto any tool call before it executes.
• Sub-agent orchestration. Spawning isolated child contexts for sub-tasks so the parent’s window doesn’t fill with exploration detail — the child returns a summary, not its full transcript.
• Error recovery. Whether the model gets a way to check its own work — tests, linters, build output — and keep iterating until the check passes, or whether it just stops the moment the output “looks done.”

That last one deserves its own sentence because it’s the highest-leverage item on the list, and Anthropic’s own guidance says so directly: without a check it can run, “looks done” is the only signal a model has. Give it something that produces a pass or fail, and the loop closes on its own.

Here’s that loop as a diagram — the shape underneath every agent harness on this list, whether it’s Claude Code, Codex CLI, or a hundred-line research prototype:

    flowchart TD
	    A["System prompt + tool schemas +\nrunning history"] --> B["Model call"]
	    B --> C{"Text only,\nor tool call?"}
	    C -->|"text only"| D["Stop — return final answer"]
	    C -->|"tool call"| E["Harness executes the tool\n(shell, file edit, search, subagent...)"]
	    E --> F["Result appended to history"]
	    F --> G{"Budget left?\n(max_turns, max_$, context room)"}
	    G -->|"yes"| B
	    G -->|"no — hit a limit"| D

Every box except “Model call” is harness code. The model only ever does one thing in this diagram: look at the current history and decide whether to speak or act. Everything about how it acts, what it’s allowed to act on, and when the loop stops is decided outside the model, by code someone wrote and tuned.

A worked example from Claude Code’s own docs makes this concrete: fixing failing tests in auth.ts takes four turns — run the test suite, read the two relevant files, apply an edit and re-run the tests, then a final text summary. That’s not “prompt in, answer out.” It’s a bounded state machine with an explicit stopping condition (no tool calls requested) and explicit escape hatches (a turn limit, a budget cap) — every one of those a harness-level decision.

A minimal loop, in actual code

Strip out the product polish and the loop above is maybe forty lines. This is close to the skeleton every harness on this list is built around, using an Anthropic-style tool-calling API:

def run_agent(client, model, system_prompt, tools, user_task, max_turns=20):
    messages = [{"role": "user", "content": user_task}]

    for turn in range(max_turns):
        response = client.messages.create(
            model=model,
            system=system_prompt,
            tools=tools,                # JSON-schema'd tool definitions
            messages=messages,
        )
        messages.append({"role": "assistant", "content": response.content})

        tool_calls = [b for b in response.content if b.type == "tool_use"]
        if not tool_calls:
            return response.content     # model produced a final text answer

        # Execute each requested tool against the real environment and feed
        # the results back in. This is the entire "agent" part of the agent.
        tool_results = []
        for call in tool_calls:
            result = execute_tool(call.name, call.input)   # harness code, not model code
            tool_results.append({
                "type": "tool_result",
                "tool_use_id": call.id,
                "content": result,
            })
        messages.append({"role": "user", "content": tool_results})

    return "stopped: max_turns exceeded"

Nothing about execute_tool is specified by the model — it’s where every harness-design decision actually lives. Compare two implementations of an edit tool for the same model, same prompt:

# Version A: hand the model a raw shell. It has to construct a working
# sed/python one-liner from scratch, with no feedback until the *next* tool
# call reveals whether it broke the file.
def execute_tool_raw_shell(name, args):
    if name == "bash":
        return subprocess.run(args["command"], shell=True,
                               capture_output=True, text=True).stdout

# Version B: SWE-agent-style bounded edit command. Shows a fixed window of
# context, re-lints immediately, and returns the error *before* the model
# moves on to something else — turning a bad edit into a one-turn fix
# instead of a compounding, multi-turn mess.
def execute_tool_bounded_edit(name, args):
    if name == "edit":
        start, end, replacement = args["start_line"], args["end_line"], args["text"]
        new_file = splice(read_file(args["path"]), start, end, replacement)
        lint_errors = run_linter(new_file)
        if lint_errors:
            return f"EDIT REJECTED — syntax errors introduced:\n{lint_errors}"
        write_file(args["path"], new_file)
        return window_view(new_file, around=start, radius=50)

Same model, same task, same underlying capability to write code. Version B just gives the model a faster, cheaper signal that it made a mistake — and a smaller, more relevant slice of the file to reason about on the next turn. That difference alone, per SWE-agent’s own ablations, is worth several points of resolve rate. The paper reports linting integration contributing roughly 3 percentage points and a structured search interface contributing several more on top of that, on a 300-issue subset — treat the exact decimals as approximate rather than load-bearing, but the direction is the paper’s consistent, central claim: interface design has a real, independently measurable effect, separate from anything about the model.

Real harnesses make genuinely different bets

It helps to see the design space by contrast rather than in the abstract. These four projects solve the same underlying problem — get a model to act in a real environment — with visibly different philosophies:

• Claude Code. A single-threaded master loop with controlled, depth-limited sub-agent spawning rather than an open swarm of parallel agents — a deliberate choice for debuggability over raw parallelism. Read-only tools (Read, Glob, Grep) can batch concurrently; state-mutating tools (Edit, Write, Bash) run one at a time to avoid conflicting writes. Permission is a first-class, explicit axis: a plan mode that never auto-edits, an accept-edits mode that auto-approves file changes but still gates Bash, and a bypass mode meant only for isolated CI containers.
• SWE-agent. The purpose-built-commands philosophy taken furthest: a custom edit command with syntax-aware bounds checking, a summarized search command instead of raw grep output, and a file viewer windowed at a tuned size (the paper reports around 100 lines as empirically strong, though the precise number-vs-alternatives comparison isn’t independently confirmable here). The thesis is that a narrow, well-designed action space beats a general-purpose shell.
• OpenHands (CodeAct). The opposite bet. Instead of a menu of bespoke JSON-schema’d tools, give the model exactly three primitives — run bash, run Python in an IPython session, and a browser DSL — and let it express arbitrary behavior as code rather than picking from a list of tool calls. The claimed justification is fewer schema-parsing and formatting errors, and better generalization than a large discrete tool menu. This is a genuine disagreement with SWE-agent’s approach, not a style preference: one camp says constrain the action space tightly and get reliability; the other says give the model a general-purpose compute primitive and let it improvise.
• Aider / Codex CLI. Aider is the cleanest illustration that harness design isn’t only about tools — it’s also about output format. Whether the model is asked to emit a diff or rewrite the whole file changes measured competence by roughly ten points on Aider’s own leaderboard, for the same model on the same task (more on the numbers below). Codex CLI, meanwhile, makes sandboxing the headline design decision: shell commands run inside a container by default, and touching anything outside it requires an explicit approval step or an opt-in --full-auto flag — permission-as-default-posture rather than permission-as-afterthought.

None of these differences are about which underlying model each project happens to plug in. They’re software engineering and interface-design decisions, made once, that then apply to whichever model sits behind them.

Context is a harness problem too

A model’s context window is fixed. A real task — explore a 200-file repo, find the relevant code, make an edit, run the tests, iterate — routinely generates far more exploration detail than fits in that window. The harness’s job is to manage that budget, and sub-agent isolation is the cleanest mechanism for it: spin up a child agent with a fresh context window for a bounded sub-task, and merge back only its final summary — not its full, messy transcript — into the parent.

    flowchart LR
	    subgraph Parent["Parent agent context"]
	        P1["Task: find and fix the\nauth bug across the repo"]
	        P2["Sub-agent 1 summary:\n'bug is in session.py:142'"]
	        P3["Sub-agent 2 summary:\n'3 call sites need the fix'"]
	        P1 --> P2 --> P3 --> PF["Parent makes the edit\nwith a small, relevant context"]
	    end
	    subgraph Sub1["Sub-agent 1: fresh context"]
	        S1a["Grep 40 files"] --> S1b["Read 6 candidates"] --> S1c["Discard exploration,\nreturn one-line summary"]
	    end
	    subgraph Sub2["Sub-agent 2: fresh context"]
	        S2a["Search call sites"] --> S2b["Read each usage"] --> S2c["Discard exploration,\nreturn one-line summary"]
	    end
	    Sub1 -.->|"summary only"| P2
	    Sub2 -.->|"summary only"| P3

The parent never sees the forty files each sub-agent grepped through — only the one-line conclusion. That’s the direct architectural answer to “the model has a fixed context window; how does the harness keep a long, exploratory task from blowing through it.” It’s also why sub-agent depth is usually capped rather than left open-ended: an unbounded tree of spawning agents makes a run hard to debug, expensive, and — per Anthropic’s own “Building Effective Agents” guidance — a cost you should earn by demonstrated improvement in outcomes, not reach for by default. Start from the simplest structure that works; add orchestration only when it measurably helps.

The numbers: same model, wildly different scores

This is the part that turns “harness matters” from a design opinion into an empirical claim worth taking seriously.

SWE-agent’s own ablation, the founding result. Holding GPT-4 Turbo fixed and giving it only a plain Linux shell instead of the custom ACI drops SWE-bench Lite performance from 18.0% to 11.0% — roughly a 7-point absolute, ~64% relative swing, from the interface alone. The full SWE-agent system resolves 12.47% of the full SWE-bench test set and 18.00% of SWE-bench Lite, against a low-single-digits resolve rate reported for the best prior non-interactive, retrieval-augmented pipeline on the same base model — treat the exact baseline percentage as approximate across sources, but the more-than-10x gap is the paper’s consistent headline claim.

A 2026 preprint on cross-harness variance (“Stop Comparing LLM Agents Without Disclosing the Harness,” arXiv:2605.23950) puts numbers on exactly the case that opened this post. I’m hedging attribution on this whole section deliberately — it’s a recent preprint I could only verify against one rendering, not cross-checked against a second independent source, though its direction is corroborated by the older, better-established SWE-agent and Aider results below. With that caveat:

• On SWE-bench Pro, the paper reports Claude Opus 4.5 scoring 45.9% under a standardized scaffold versus 55.4% under the Claude Code harness — a 9.5-point swing, same weights.
• On SWE-bench Verified, it cites Holistic Agent Leaderboard (HAL) numbers showing Claude Sonnet 4.5 going from 68% under a SWE-Agent-style scaffold to 34% under a “HAL Generalist” scaffold — the 34-point gap this post opened with — and GPT-5 Medium moving 46% to 12%, also a 34-point gap.
• On Terminal-Bench 2.0, holding the model fixed and changing only the harness is reported to move pass@1 from 69.7% to 77.0% — a 7.3-point gain from scaffolding alone.
• The paper’s own controlled factorial experiment — 3 frontier models × 3 harnesses on a 100-task SWE-bench Verified subset — reports average harness-induced variance at roughly 7.8x average model-induced variance: changing harness moved scores 8.5–13.0 points, changing model (fixed harness) moved scores only 2.5–5.0 points. Most strikingly, 6 of 9 possible model-ranking comparisons reversed depending on which harness was used — the harness didn’t just add a fixed offset to every model’s score, it flipped which model looked better.

Aider’s edit-format leaderboard is the cleanest illustration precisely because it isolates one narrow variable — not agentic-loop design, not tool selection, just how the model is asked to express an edit. Gemini-exp-1206 scores 80.5% using a “whole file rewrite” format versus 69.2% using a “diff” format — an 11.3-point gap from format alone, same model. o1-mini shows a similar 9.6-point gap (70.7% vs 61.1%). Whole-file rewrites are easier for a model to produce correctly but far more token-expensive and don’t scale to large files; diffs are cheap but harder to emit without error. Aider’s own maintainers pick the format per-model, which is itself a harness decision quietly shaping every reported score on that leaderboard.

Devin, and the clearest public demonstration of this whole phenomenon. Cognition reported Devin resolving 13.86% of SWE-bench issues end-to-end against a then-cited unassisted baseline of 1.96% — a widely publicized jump. The comparison drew documented criticism: Devin was evaluated in a more permissive “agent setting” than the baselines it was compared against, was allowed up to 45 minutes of iteration per task, and its evaluation subset only overlapped the original paper’s subset in 140 instances. Within weeks, the open-source SWE-agent — a harness, not a new model — exceeded Devin’s reported number using off-the-shelf GPT-4. A better scaffold on a generic model beat a bespoke product’s model-plus-harness combination. That’s usually the single clearest evidence that a leaderboard number without harness disclosure is not a model-capability fact — it’s a joint function of model, harness, and often-undisclosed settings like temperature or tool-calling API version. (SWE-bench Verified’s own leaderboard now standardizes on a minimal reference harness for exactly this reason — and even notes that results aren’t fully comparable across its own harness’s versions, which is itself a small demonstration of how sensitive these numbers are to scaffold details.)

One more caveat worth stating plainly since it affects how to read the original, non-Verified SWE-bench number above: a widely cited critique argues a meaningful fraction of “successes” on the original benchmark involved either the fix leaking into the issue text itself or passing due to inadequate test coverage rather than a genuinely correct patch. I haven’t independently verified that critique’s exact figures against a primary source, so treat it as a well-known caveat rather than a settled statistic — but it’s exactly the class of problem that motivated building SWE-bench Verified’s smaller, human-reviewed 500-task subset in the first place.

What a harness engineer actually does

None of the work above is prompt engineering, and very little of it touches model weights. A model or research engineer’s day is centered on the model itself: training runs, architecture, loss curves, data mixture, post-training, evals of the model in isolation. A harness engineer’s day looks more like systems and HCI work aimed at a model that’s already fixed and treated as a black box:

• Designing the action space — which tools exist, what their inputs and outputs look like, and crucially what feedback they return when something goes wrong (the SWE-agent linting example: a rejected edit with a clear error is worth more than a silently-applied bad edit).
• Managing the context budget — what’s in the system prompt every turn, when to compact, when to spin off a sub-agent instead of letting the parent’s window fill with exploration noise.
• Building the sandbox and permission model — what a tool call is allowed to touch, whether it needs approval, and what the blast radius of a mistake looks like.
• Instrumenting verification loops — wiring in tests, linters, and build output so the agent has something better than “looks done” to check against.
• Running the evals that actually isolate harness variables — which, per the numbers above, is genuinely hard to do honestly, because it requires disclosing and controlling for the scaffold, not just the model.

It’s tempting to read this as glue code — plumbing around the “real” work happening inside the model. The numbers argue against that framing directly: a single added search sub-agent tool has been reported to add multiple points on SWE-bench Pro across several different underlying models; a bounded edit interface with better error feedback is worth several points in a controlled ablation; harness choice alone reversed model rankings in most of a controlled study’s comparisons. This is applied systems and interface-design work with a measurable, sometimes larger-than-a-model-upgrade impact on what actually gets accomplished — closer in spirit to compiler or IDE design than to what “prompt tweaking” implies.

Where this breaks down

Harness complexity is a cost, not a default. More tools, more sub-agents, more orchestration is not automatically better — it’s more context spent on scaffolding, more latency, more surface area for something to go wrong. Anthropic’s own guidance is explicit that the simplest structure that works should be the starting point, with complexity added only when it demonstrably improves measured outcomes.

A single benchmark number is a confound, not a fact about the model. Every result in this post is (model, harness, and often undisclosed inference settings) collapsed into one percentage. Treating “Model X scores Y% on SWE-bench Verified” as a pure model-capability statement is, per the evidence above, the most common error in casual coverage of these benchmarks — and it’s an error the field’s own leaderboards are starting to correct for by standardizing on reference harnesses.

Some of the sharpest numbers here are recent and not independently re-verified. The 2026 cross-harness disclosure paper’s specific percentages (45.9/55.4, 68/34, 46/12, 69.7/77.0, the 7.8x variance ratio) are corroborated in direction by the older, better-established SWE-agent and Aider results, but I could only check them against one rendering of a preprint. Read them as “a 2026 paper analyzing harness disclosure reports,” not as settled consensus figures the way SWE-bench Verified’s own methodology is.

A better harness doesn’t create capability the model doesn’t have. It removes friction — bad interfaces, wasted context, no feedback signal, unnecessary approval stalls — that was preventing the model from using the capability it already had. Ceiling still comes from the model; how close a given run gets to that ceiling comes from the harness. Confusing the two in either direction is the mistake this whole post is arguing against.

The practical upshot: if you’re building or evaluating an agent and the numbers look disappointing, the model swap is not necessarily the highest-leverage next experiment. Given a 7.8x variance ratio between harness and model choice in at least one controlled study, the interface, the context strategy, and the verification loop are very often where the actual points are sitting.