The leaderboard said five architectures tied. The traces said four of them never ran as designed.
The first run set measured 90 runs of five agent architectures on Sonnet 5. Every architecture scored between 0.980 and 1.000, and 85 of the 90 runs were recorded with no failure at all. We read the traces before publishing it. This page is what they showed, with every count taken from files in the repository that anyone can open.
Retracted
That run set is withdrawn. Its runs stay in the repository exactly as recorded (results/2026-07-25T20-18-35Z-3fs5), as evidence. Nothing on this site ranks its architectures or draws a conclusion from its scores.
72/72
plan, decompose and select steps that ended in an error
35/36
executors handed an empty plan
0/18
orchestrator runs that split the task
36/36
second and third proposers that started with no tools
0/18
verified runs sent back for revision
70
times the guard denied the harness’s own tool
What the leaderboard said
A tie, and a tidy one.
Five architectures, nine tasks, two repeats. The scores sat so close together that no interval was needed to see they overlapped, the failure column was nearly empty, and the verdict wrote itself: none of the extra structure earned back its cost. It matched what everyone already believed, which is the moment to be most careful.
Architecture0.900.951.00Mean scoreCost per run
Single0.990 · $0.050 per run0.990$0.050
Plan-execute0.990 · $0.070 per run0.990$0.070
Orchestrator-workers0.980 · $0.089 per run0.980$0.089
Plan-execute-verify0.990 · $0.109 per run0.990$0.109
Propose-select1.000 · $0.187 per run1.000$0.187
Mean score over the 18 runs of each architecture. The axis starts at 0.90, not at zero: on a full axis the five dots are one dot.
What the traces said
Three faults in our harness, none in the models.
Every step of every run was recorded as a trace. Reading them showed that the designs were not the ones being compared. No run crashed, so nothing looked wrong in a table.
1What the traces showed
The sandbox refused the structured reply
When an architecture asks the model for a structured reply (a plan, a list of subtasks, a choice of proposal), the Agent SDK has the model call a tool named StructuredOutput. Our sandbox guard denies any tool outside the task’s toolset, and nobody had told it about this one. The reply was refused, the step had a single turn to recover in, and it ended in an error with nothing to hand on.
The next step ran anyway. Executors received an empty plan; the orchestrator, with no decomposition, fell back to one worker given the whole task; the selector never returned a choice, so the first proposal stood in for it. The two steps the guard did not stop failed the same way, on a validation error from the same tool.
72/72
plan, decompose and select steps ended in an error
70
denials by the guard, every one of them of StructuredOutput
35/36
executors were handed <plan></plan>, empty
0/18
orchestrator runs split the task
2What the traces showed
Parallel agents shared one tool connection
The task’s tools were served by one in-process server object, handed to every model call. A server can serve one connection at a time. Sequential steps never noticed, because each connection closed before the next began. Parallel steps raced for it: the first proposer got the tools, and the other two started without them.
“Since I only have read-only access, I was not given any actual tool schemas to invoke in this session, so I cannot report live data pulled from the system.”
The same sessions listed Gmail, Calendar and Drive connectors from the operator’s own account. They were unauthenticated and inert, but the claim that a run was shown exactly the task’s tools and nothing else was not true.
36/36
second and third proposers started with none of the task’s tools
0
tool calls made by those proposers
70/252
sessions also listed claude.ai connectors from the operator’s own login
3What the traces showed
The verifier could not say no
The verifier was never asked for a structured verdict, and when its prose could not be read as one the code treated it as approval. The revise loop could never run. The one run that did fail its scorer was approved with a list of checks that all passed, including a restock that the task’s own policy forbids (“Never restock a line you have merely refunded”).
“All checks confirm the executor’s work is correct and complete.”
Four of five were a single agent in a trench coat.
Put the three faults together and each architecture reduces to something simpler than its name. Crossed-out steps never did their job in the runs recorded; the counts beside them are read off the run records.
Designed → as it ranWhat the run records showScored
→
Singleran as designed
The control: one agent, nothing to skip. It ran as designed in 18/18 runs.
Scored0.990
→
Plan-executedid not run as designed
18/18runs: the planning phase ended in an error
Scored0.990
→
Orchestrator-workersdid not run as designed
18/18runs: the decomposition phase ended in an error
18/18runs: the task went to a single worker
Scored0.980
→
Plan-execute-verifydid not run as designed
18/18runs: the planning phase ended in an error
18/18runs: the verifier sent nothing back for revision, including the 1 that scored below 1.0
Scored0.990
→
Propose-selectdid not run as designed
36/54proposals made without a single tool call
18/18runs: the selection phase ended in an error
Scored1.000
Watch it happen
The same task, replayed from the recorded traces. Pick propose-select and watch the second and third proposers: they start, make no tool call, and finish. Pick plan-execute and watch the planner end in an error before the executor starts.
A step that errors does not fail the run: the next step still has the task. The selector failed, so proposal 1 won, and proposal 1 was the only one with tools. The planner failed, so the executor did what a single agent does. The score was the final agent’s score, and the final agent was a single agent. Every number on the leaderboard was real; the conclusion it invited was about designs that did not exist.
$2.43
of $9.11 spent
27% of the money went into 108 steps that errored or ran blind. Together they are 53% of what the other four architectures cost over single.
Single$0.050 · none wasted
Plan-execute$0.070 · 27% wasted
Orchestrator-workers$0.089 · 22% wasted
Plan-execute-verify$0.109 · 12% wasted
Propose-select$0.187 · 45% wasted
steps that did their jobsteps that errored, or proposed with no tool to look anything up
Why 437 passing tests did not catch it
The strategy tests replaced the model call with a fake that returned a structured reply whether or not the code had asked for one, so a planner that never got its reply could not fail. The only check against the real SDK started a single session, so a second one could not lose its tools. Nothing asked the question that mattered: did the architecture actually run?
What changed
A run has to prove its design ran.
The guard now knows about the structured-output tool, and every model call gets its own tool server. A verifier that returns no readable verdict is no longer an approval. More to the point, every architecture now declares what running as designed looks like (a plan with at least one step, a fan-out that happened, a verdict that was read, a selector that chose), and the runner adds checks of its own: no harness tool denied, every tool server connected, no foreign ones, no silent fallback. A run that fails a check is excluded from every ranking and counted, never scored.
The checks are listed one by one, with their pass rates, on the harness health page, and the statistics that rank what is left are on the methodology page. In the run set published here, 900 of 900 runs passed every check and were ranked.
Check it yourself
Every count above is a one-liner.
The run records are plain JSON and stay in the repository unchanged. From the repository root:
cd results/2026-07-25T20-18-35Z-3fs5
# steps that asked for a structured reply and ended in an error (72)
jq -s '[.[] | .phases[] | select(.phase | test("^(plan|decompose|select)$")) | select(.error)] | length' runs/*.json
# guard denials, over the whole set (70)
jq -s '[.[] | .deniedToolCalls] | add' runs/*.json
# orchestrator runs that never gave more than one worker a task (18 of 18)
jq -s '[.[] | select(.strategyId == "orchestrator-workers") | [.phases[] | select(.phase | startswith("worker"))] | length | select(. <= 1)] | length' runs/*.json
# second and third proposers that made no tool call (36)
jq -s '[.[] | select(.strategyId == "debate-select") | .phases[] | select((.phase | test("^propose-[23]$")) and .toolCalls == 0)] | length' runs/*.json
The counts that need the traces (empty plans, the connectors, the proposers’ tool lists) were taken from the 90 original traces, which this build carries under results/2026-07-25T20-18-35Z-3fs5/traces/. apps/web/test/postmortem.test.ts re-derives them from those traces on every test run, and fails if a number on this page is wrong. The run set was recorded at 164f958, where 437 tests passed.
What we took from it
Before you compare agent architectures, prove each one actually ran.
A plausible score is not evidence that a design was measured, and an agreeable result is the one to distrust. The corrected benchmark runs every architecture on three Claude models, excludes any run that did not run as designed, and reports what the structure buys with an interval on every number.