ScaffoldBench

900 pre-registered runs · 6 architectures · 3 models

Scale beat scaffold.

We ran planners, verifiers, worker pools, selectors and review gates against a plain single agent on Haiku 4.5, Sonnet 5.5 and Opus 5.5. On no model did any scaffold measurably beat the single agent, and every one cost 1.3 to 4.8 times as much per task. One step up in model bought more score than any scaffold did.

See the results Pre-registration scorecard
scaffolds that beat a single agent
0 / 15
5 architectures × 3 models, paired by task, Holm-adjusted
what a scaffold costs per task
×1.3–4.8
15 of 15 shown costlier than the single agent
score from one model step up
+0.13
Haiku 4.5 → Sonnet 5.5, single agent, at ×1.24 the cost
from the best Haiku 4.5 scaffold
+0.04
Propose-select, not separable, at ×4.34 the cost
Same task, 6 harnesses: Cancel a fare the customer wrongly believes is refundable to their cardReplay · Haiku 4.5 · repeat 1
0.0s · 8× speed

  1. Single

    $0

  2. Plan-execute

    $0

  3. Orchestrator-workers

    $0

  4. Plan-execute-verify

    $0

  5. Propose-select

    $0

  6. Gated-execute

    $0

The result · scaffold-or-scale-v1

Within a model, the scaffolds scatter around the plain agent. Across models, the plain agent climbs.

Five scaffolds against a single agent on three Claude models: 15 paired comparisons, none separated in the scaffold's favour. On Haiku 4.5, four of the five scored below the agent they wrap.

SinglePlan-executeOrchestrator-workersPlan-execute-verifyPropose-selectGated-execute
0.500.550.600.650.700.750.800.850.900.951.00scoreHaiku 4.5single agent $0.091 / runSonnet 5.5single agent $0.116 / runOpus 5.5single agent $0.252 / runSingle on Haiku 4.5: 0.771 [0.673, 0.868], $0.091 per runPlan-execute on Haiku 4.5: 0.754 [0.634, 0.862], $0.128 per runOrchestrator-workers on Haiku 4.5: 0.701 [0.573, 0.824], $0.171 per runPlan-execute-verify on Haiku 4.5: 0.723 [0.588, 0.845], $0.305 per runPropose-select on Haiku 4.5: 0.809 [0.711, 0.901], $0.373 per runGated-execute on Haiku 4.5: 0.677 [0.534, 0.809], $0.862 per runSingle on Sonnet 5.5: 0.900 [0.843, 0.947], $0.116 per runPlan-execute on Sonnet 5.5: 0.933 [0.895, 0.966], $0.146 per runOrchestrator-workers on Sonnet 5.5: 0.924 [0.883, 0.962], $0.216 per runPlan-execute-verify on Sonnet 5.5: 0.930 [0.856, 0.985], $0.278 per runPropose-select on Sonnet 5.5: 0.939 [0.903, 0.972], $0.437 per runGated-execute on Sonnet 5.5: 0.927 [0.889, 0.963], $0.495 per runSingle on Opus 5.5: 0.939 [0.903, 0.974], $0.252 per runPlan-execute on Opus 5.5: 0.947 [0.913, 0.977], $0.336 per runOrchestrator-workers on Opus 5.5: 0.921 [0.869, 0.961], $0.459 per runPlan-execute-verify on Opus 5.5: 0.947 [0.890, 0.986], $0.561 per runPropose-select on Opus 5.5: 0.958 [0.925, 0.985], $0.928 per runGated-execute on Opus 5.5: 0.956 [0.922, 0.985], $0.934 per runOne model up: +0.129at ×1.24 the costBest Haiku 4.5 scaffold: +0.038at ×4.34 the cost, not separable
0.500.550.600.650.700.750.800.850.900.951.00scoreHaiku 4.5$0.091Sonnet 5.5$0.116Opus 5.5$0.252Single on Haiku 4.5: 0.771 [0.673, 0.868], $0.091 per runPlan-execute on Haiku 4.5: 0.754 [0.634, 0.862], $0.128 per runOrchestrator-workers on Haiku 4.5: 0.701 [0.573, 0.824], $0.171 per runPlan-execute-verify on Haiku 4.5: 0.723 [0.588, 0.845], $0.305 per runPropose-select on Haiku 4.5: 0.809 [0.711, 0.901], $0.373 per runGated-execute on Haiku 4.5: 0.677 [0.534, 0.809], $0.862 per runSingle on Sonnet 5.5: 0.900 [0.843, 0.947], $0.116 per runPlan-execute on Sonnet 5.5: 0.933 [0.895, 0.966], $0.146 per runOrchestrator-workers on Sonnet 5.5: 0.924 [0.883, 0.962], $0.216 per runPlan-execute-verify on Sonnet 5.5: 0.930 [0.856, 0.985], $0.278 per runPropose-select on Sonnet 5.5: 0.939 [0.903, 0.972], $0.437 per runGated-execute on Sonnet 5.5: 0.927 [0.889, 0.963], $0.495 per runSingle on Opus 5.5: 0.939 [0.903, 0.974], $0.252 per runPlan-execute on Opus 5.5: 0.947 [0.913, 0.977], $0.336 per runOrchestrator-workers on Opus 5.5: 0.921 [0.869, 0.961], $0.459 per runPlan-execute-verify on Opus 5.5: 0.947 [0.890, 0.986], $0.561 per runPropose-select on Opus 5.5: 0.958 [0.925, 0.985], $0.928 per runGated-execute on Opus 5.5: 0.956 [0.922, 0.985], $0.934 per run
Dots: mean score over 25 tasks (2 repeats each, 50 runs per dot); whiskers: 95% task-clustered bootstrap interval. The ring is the plain single agent; the dashed line carries its score across the column, so a dot above it is a scaffold that scored higher on average, though none did by a margin the paired tests could separate.
Show the numbers
ModelArchitectureScore95% CICost / runvs single (Holm)
Haiku 4.5Single0.771[0.673, 0.868]$0.091control
Haiku 4.5Plan-execute0.754[0.634, 0.862]$0.128−0.017 · inconclusive
Haiku 4.5Orchestrator-workers0.701[0.573, 0.824]$0.171−0.070 · inconclusive
Haiku 4.5Plan-execute-verify0.723[0.588, 0.845]$0.305−0.048 · inconclusive
Haiku 4.5Propose-select0.809[0.711, 0.901]$0.373+0.038 · inconclusive
Haiku 4.5Gated-execute0.677[0.534, 0.809]$0.862−0.094 · inconclusive
Sonnet 5.5Single0.900[0.843, 0.947]$0.116control
Sonnet 5.5Plan-execute0.933[0.895, 0.966]$0.146+0.034 · inconclusive
Sonnet 5.5Orchestrator-workers0.924[0.883, 0.962]$0.216+0.025 · inconclusive
Sonnet 5.5Plan-execute-verify0.930[0.856, 0.985]$0.278+0.031 · inconclusive
Sonnet 5.5Propose-select0.939[0.903, 0.972]$0.437+0.040 · inconclusive
Sonnet 5.5Gated-execute0.927[0.889, 0.963]$0.495+0.028 · inconclusive
Opus 5.5Single0.939[0.903, 0.974]$0.252control
Opus 5.5Plan-execute0.947[0.913, 0.977]$0.336+0.008 · inconclusive
Opus 5.5Orchestrator-workers0.921[0.869, 0.961]$0.459−0.019 · inconclusive
Opus 5.5Plan-execute-verify0.947[0.890, 0.986]$0.561+0.008 · inconclusive
Opus 5.5Propose-select0.958[0.925, 0.985]$0.928+0.018 · inconclusive
Opus 5.5Gated-execute0.956[0.922, 0.985]$0.934+0.016 · inconclusive

Read this before quoting it

  • 25 tasks are 25 clusters. Power is set by the task count, not the number of runs, so most contrasts are inconclusive rather than shown equivalent: “no scaffold won” is not “every scaffold is useless”.
  • Opus 5.5 had little headroom. Its single agent already scored 0.939, and on 10 of the 25 tasks every architecture scored 0.99 or more, leaving no room to show a gain.
  • We wrote the pre-registration. It was committed and tagged before the runs, and the single pilot and what it changed are disclosed in it.
  • Our first leaderboard was wrong. Four of five designs never ran as designed; every run now has to prove it did. Read the post-mortem.

Pre-registration scorecard

We wrote down six predictions before the first run. Half of them failed.

The hypotheses, the analysis and its settings were committed and tagged prereg-v1 before any of the 900 runs. Each verdict below is recomputed from the run records by the same code, with Holm adjustment within each family.

  • ✓ 1 supported
  • ≈ 1 consistent
  • ◐ 1 partial
  • ✗ 3 not supported
  1. H1Not supported

    Confirmatory

    Scaffolds help the small model

    Predicted: On Haiku 4.5, plan-execute-verify or gated-execute scores better than a single agent.

    Neither separated, and both point estimates are below zero: Plan-execute-verify −0.048 [−0.171, +0.051], Gated-execute −0.094 [−0.215, +0.005].

  2. H2Consistent

    Confirmatory

    Scaffolds stop paying on the large model

    Predicted: On Opus 5.5, no architecture scores better than a single agent.

    None of 5 is better. None is shown equivalent either, so the data cannot say the gain is under 0.05.

  3. H3Supported

    Confirmatory

    Every scaffold costs more

    Predicted: On every model, every multi-call architecture costs more per task than a single agent.

    15 of 15 costlier, each interval above ×1: ×1.34 to ×4.77 the cost of a run.

  4. H4Not supported

    Secondary

    The gain shrinks as models scale

    Predicted: Plan-execute-verify gains more over a single agent on Haiku 4.5 than on Opus 5.5.

    Difference in gains −0.056 [−0.181, +0.050]: inconclusive, and the wrong sign.

  5. H5Partial

    Secondary

    Scale beats scaffold, one step up

    Predicted: A single agent on Sonnet 5.5 scores better than every architecture on Haiku 4.5.

    3 of 6 shown below Sonnet’s single agent (Orchestrator-workers, Plan-execute-verify, Gated-execute); the other 3 are below it on point estimates but not separated after Holm. None matches it while costing less.

  6. H6Not supported

    Descriptive only

    Fan-out buys time on breadth research

    Predicted: On the gold-key research tasks, orchestrator-workers ties a single agent on score and is faster.

    Three tasks, so no verdict. Orchestrator-workers was slower on 3 of 3 models (median 71 s vs 69 s; 44 s vs 27 s; 64 s vs 34 s).

H1–H3 are confirmatory (the baseline family); H4 and H5 are secondary, each its own family; H6 has three tasks, fewer than the five a verdict needs, so it is described rather than tested. The pre-registration also discloses the pilot and the two changes it led to. Every number: REPORT.md.

One thing worth a second lookExploratory · not pre-registered

Plan-execute on Sonnet 5.5 matched a single agent on Opus 5.5, at ×0.64 the cost.

The planner did not separably beat Sonnet 5.5's own single agent (+0.034, not separable). What it did was close enough of the gap to Opus 5.5 for the equivalence test to pass: within ±0.05 of Opus 5.5 working alone, on the same tasks and repeats. Sonnet 5.5's single agent is cheaper still (×0.48) but scored −0.040 against it, not separable. The comparison comes from the analysis's scale family, computed but not predicted, so read it as a lead for the next run set rather than a finding of this one.

Score difference
−0.006
95% CI [−0.033, +0.020]
Verdict
equivalent
within ±0.05, two one-sided tests
Cost ratio
×0.64
paired, geometric mean per cell
Pairs
50
25 tasks, paired by task and repeat

Read the traces

The averages hide the runs. Here are a few, call by call.

Every prompt, tool call, review and verdict is recorded. The repository keeps 9 of the 900 traces, chosen to show a design at work; every run's score, phases and final answer are on its page.

Replay the six-way race above, call by call →A revision that did not help →The control, missing a step →

The evidence, model by model

Each scaffold against its own single agent: what it gained, and what it cost.

Every architecture against a plain single agent on the same model, the same tasks and the same repeats. Up the page is score gained; across it, what that cost. An interval wholly above the grey band is structure that bought real score.

None of the 15 architecture–model combinations scores reliably above a single agent. All 15 are equivalent to it or not separable at this sample size.

single, the control (1×, 0)equivalent to single (±0.05)Plan-executeOrchestrator-workersPlan-execute-verifyPropose-selectGated-execute
Haiku 4.550 paired runs each−0.20−0.100+0.101×1.5×3×5×8×cost relative to single (log scale)Δ scorePlan-execute on Haiku 4.5: −0.017 score [−0.143, +0.084] at 1.5× the cost of single [1.4×, 1.7×], not separable from single (50 paired runs); too few tasks carry the spread to show equivalence.Orchestrator-workers on Haiku 4.5: −0.070 score [−0.152, −0.001] at 2.0× the cost of single [1.6×, 2.5×], not separable from single (50 paired runs); too few tasks carry the spread to show equivalence.Plan-execute-verify on Haiku 4.5: −0.048 score [−0.171, +0.051] at 3.3× the cost of single [2.9×, 3.9×], not separable from single (50 paired runs); too few tasks carry the spread to show equivalence.Propose-select on Haiku 4.5: +0.038 score [−0.018, +0.094] at 4.3× the cost of single [4.0×, 4.7×], not separable from single (50 paired runs).Gated-execute on Haiku 4.5: −0.094 score [−0.215, +0.005] at 4.8× the cost of single [2.9×, 7.7×], not separable from single (50 paired runs); too few tasks carry the spread to show equivalence.singlePlan-executeOrchestrator-workersPlan-execute-verifyPropose-selectGated-execute
Sonnet 5.550 paired runs each−0.20−0.100+0.101×1.5×3×5×8×cost relative to single (log scale)Δ scorePlan-execute on Sonnet 5.5: +0.034 score [0.000, +0.087] at 1.3× the cost of single [1.3×, 1.4×], not separable from single (50 paired runs); too few tasks carry the spread to show equivalence.Orchestrator-workers on Sonnet 5.5: +0.025 score [−0.005, +0.078] at 1.8× the cost of single [1.5×, 2.2×], not separable from single (50 paired runs); too few tasks carry the spread to show equivalence.Plan-execute-verify on Sonnet 5.5: +0.031 score [−0.019, +0.077] at 2.4× the cost of single [2.2×, 2.6×], not separable from single (50 paired runs); too few tasks carry the spread to show equivalence.Propose-select on Sonnet 5.5: +0.040 score [+0.003, +0.095] at 4.1× the cost of single [3.8×, 4.3×], not separable from single (50 paired runs); too few tasks carry the spread to show equivalence.Gated-execute on Sonnet 5.5: +0.028 score [−0.012, +0.083] at 2.6× the cost of single [2.0×, 3.5×], not separable from single (50 paired runs); too few tasks carry the spread to show equivalence.singlePlan-executeOrchestrator-workersPlan-execute-verifyPropose-selectGated-execute
Opus 5.550 paired runs each−0.20−0.100+0.101×1.5×3×5×8×cost relative to single (log scale)Δ scorePlan-execute on Opus 5.5: +0.008 score [−0.020, +0.039] at 1.4× the cost of single [1.3×, 1.6×], not separable from single (50 paired runs); too few tasks carry the spread to show equivalence.Orchestrator-workers on Opus 5.5: −0.019 score [−0.051, +0.006] at 1.8× the cost of single [1.5×, 2.2×], not separable from single (50 paired runs); too few tasks carry the spread to show equivalence.Plan-execute-verify on Opus 5.5: +0.008 score [−0.042, +0.053] at 2.4× the cost of single [2.1×, 2.7×], not separable from single (50 paired runs); too few tasks carry the spread to show equivalence.Propose-select on Opus 5.5: +0.018 score [0.000, +0.043] at 4.1× the cost of single [3.7×, 4.5×], not separable from single (50 paired runs); too few tasks carry the spread to show equivalence.Gated-execute on Opus 5.5: +0.016 score [0.000, +0.041] at 2.5× the cost of single [1.9×, 3.2×], not separable from single (50 paired runs); too few tasks carry the spread to show equivalence.singlePlan-executeOrchestrator-workersPlan-execute-verifyPropose-selectGated-execute
Vertical whiskers: interval on the score difference. Horizontal whiskers: interval on the cost multiple. A point without a whisker has no interval, and the table says why. Grey band: ±0.05, the equivalence margin. Every number is a mean over tasks, not runs: whole tasks are resampled (a cluster bootstrap of 10,000 replicates), so a task's repeats stay together and the 95% intervals are as wide as the tasks make them. Architectures are compared in pairs on the cells both ran, the same task and the same repeat: score differences are differences of means, cost and latency are geometric-mean ratios, and for them “better” means cheaper or faster. Each verdict comes from tests, not from whether two intervals overlap, and is Holm-adjusted within its family (one kind of contrast on one metric, across every model) to hold the chance of any false call in a family at 5%; so a verdict can read “not separable” while the unadjusted interval excludes zero. “Equivalent” means two one-sided tests show the difference within ±0.05 of score (×/÷1.1 for cost and latency). With fewer than 5 tasks no verdict is drawn, and where every task moved by the same amount there is no interval, which the table says instead of printing a zero-width one.
Show the numbers
ModelArchitectureΔ scoreintervalCost ×intervalVerdict
Haiku 4.5Plan-execute−0.017−0.143 to +0.0841.5×1.4×–1.7×inconclusive (too few tasks carry the spread to show equivalence)
Haiku 4.5Orchestrator-workers−0.070−0.152 to −0.0012.0×1.6×–2.5×inconclusive (too few tasks carry the spread to show equivalence)
Haiku 4.5Plan-execute-verify−0.048−0.171 to +0.0513.3×2.9×–3.9×inconclusive (too few tasks carry the spread to show equivalence)
Haiku 4.5Propose-select+0.038−0.018 to +0.0944.3×4.0×–4.7×inconclusive
Haiku 4.5Gated-execute−0.094−0.215 to +0.0054.8×2.9×–7.7×inconclusive (too few tasks carry the spread to show equivalence)
Sonnet 5.5Plan-execute+0.0340.000 to +0.0871.3×1.3×–1.4×inconclusive (too few tasks carry the spread to show equivalence)
Sonnet 5.5Orchestrator-workers+0.025−0.005 to +0.0781.8×1.5×–2.2×inconclusive (too few tasks carry the spread to show equivalence)
Sonnet 5.5Plan-execute-verify+0.031−0.019 to +0.0772.4×2.2×–2.6×inconclusive (too few tasks carry the spread to show equivalence)
Sonnet 5.5Propose-select+0.040+0.003 to +0.0954.1×3.8×–4.3×inconclusive (too few tasks carry the spread to show equivalence)
Sonnet 5.5Gated-execute+0.028−0.012 to +0.0832.6×2.0×–3.5×inconclusive (too few tasks carry the spread to show equivalence)
Opus 5.5Plan-execute+0.008−0.020 to +0.0391.4×1.3×–1.6×inconclusive (too few tasks carry the spread to show equivalence)
Opus 5.5Orchestrator-workers−0.019−0.051 to +0.0061.8×1.5×–2.2×inconclusive (too few tasks carry the spread to show equivalence)
Opus 5.5Plan-execute-verify+0.008−0.042 to +0.0532.4×2.1×–2.7×inconclusive (too few tasks carry the spread to show equivalence)
Opus 5.5Propose-select+0.0180.000 to +0.0434.1×3.7×–4.5×inconclusive (too few tasks carry the spread to show equivalence)
Opus 5.5Gated-execute+0.0160.000 to +0.0412.5×1.9×–3.2×inconclusive (too few tasks carry the spread to show equivalence)

How we got here · Act 1: the harness that lied

The leaderboard said five architectures tied. The traces said four of them never ran as designed.

The first published run set, 90 runs on Sonnet 5 in July 2026, scored every architecture between 0.980 and 1.000. Reading the traces showed why it looked like a tie: 4 of the 5 designs never executed as designed, so what was measured was mostly a single agent with dead steps around it. No run crashed, and every score looked plausible.

  1. Singleran as designed

    The control: one agent, nothing to skip. It ran as designed in 18/18 runs.

    Scored0.990

  2. Plan-executedid not run as designed

    • 18/18runs: the planning phase ended in an error
    Scored0.990

  3. Orchestrator-workersdid not run as designed

    • 18/18runs: the decomposition phase ended in an error
    • 18/18runs: the task went to a single worker
    Scored0.980

  4. Plan-execute-verifydid not run as designed

    • 18/18runs: the planning phase ended in an error
    • 18/18runs: the verifier sent nothing back for revision, including the 1 that scored below 1.0
    Scored0.990

  5. Propose-selectdid not run as designed

    • 36/54proposals made without a single tool call
    • 18/18runs: the selection phase ended in an error
    Scored1.000

The fault was in our harness, not the models. The sandbox guard denied the tool the Agent SDK uses for structured output, so planning, decomposition and selection steps could not hand back their result, and with a single turn they had no way to recover. The guard recorded denials in 70 of 90 runs.

So a run now has to prove that its architecture executed: each design declares checks such as “the plan was not empty” or “the work fanned out”, and a run that fails one is excluded from every ranking on this site and counted instead. The records behind every number above stay in the repository, unchanged, as evidence: results/2026-07-25T20-18-35Z-3fs5.

Read the post-mortem

Why you can trust this

Trust, but verify: each claim links to the code that enforces it.

  • Proof the design ran

    Each architecture declares what running as designed looks like: a non-empty plan, a real fan-out, a parsed verdict. A run that fails a check is excluded from rankings and counted, never scored.

    packages/core/src/integrity.ts

  • Same inputs for every design

    Every architecture gets the same model, tools, task prompt, domain policy and budget. Only the harness around the model changes.

    packages/core/src/systemPrompt.ts

  • Scored by state, not by vibes

    Tool-use tasks are graded by checking the final state of the world and the journal of tool calls: no language model decides who won. Research tasks are graded by a judge model against a rubric, and say so.

    packages/tasks/src/defineToolUseTask.ts

  • A sandbox that says no

    A guard denies any tool outside the task’s toolset, before it runs, and every denial is written to the trace.

    packages/core/src/guard.ts

  • Paired, task-clustered statistics

    Architectures are compared on the same task and repeat, bootstrapped over tasks so repeats are not mistaken for independent evidence.

    packages/core/src/analysis.ts

  • Outages are not failures

    A run lost to the API is never scored as the architecture failing. It is excluded from every ranking and reported in the health count.

    packages/core/src/runner.ts

Explore

Go as deep as the data goes.