ScaffoldBench

Results · Claude Sonnet 5.5

Every architecture on Sonnet 5.5

Ranked by paired comparison: an architecture ranks below another only when, task by task and repeat by repeat, it scores reliably lower. Ties are shown as ties. Every number carries an interval bootstrapped over tasks, or says why it has none.

Runs
300
recorded on this model
Ranked
300
design verified, no API loss
Excluded: fidelity
0
the design did not run as designed
Excluded: API
0
lost to the API after retries
Retried
5
needed a second attempt

Harness health, check by check →

On Sonnet 5.5, no architecture scores reliably above a single agent. 5 are not separable from it at this sample size.

Leaderboard

  1. Rank 1, shared. Not separable from any other architecture.Propose-select0.9390.903–0.972
    Pass
    62% (43%–78%)
    Cost / run
    $0.437 ($0.337–$0.538)
    Latency
    1m 14s (58.2–1m 30s)
    Runs
    50/50
    vs single+0.040not separablefew informative tasks4.1× the cost
  2. Rank 1, shared. Not separable from any other architecture.Plan-execute0.9330.895–0.966
    Pass
    58% (40%–74%)
    Cost / run
    $0.146 ($0.113–$0.182)
    Latency
    45.8s (36.9–55.3s)
    Runs
    50/50
    vs single+0.034not separablefew informative tasks1.3× the cost
  3. Rank 1, shared. Not separable from any other architecture.Plan-execute-verify0.9300.856–0.985
    Pass
    74% (56%–87%)
    Cost / run
    $0.278 ($0.207–$0.353)
    Latency
    1m 24s (1m 3s–1m 47s)
    Runs
    50/50
    vs single+0.031not separablefew informative tasks2.4× the cost
  4. Rank 1, shared. Not separable from any other architecture.Gated-execute0.9270.889–0.963
    Pass
    58% (39%–75%)
    Cost / run
    $0.495 ($0.277–$0.736)
    Latency
    2m 34s (1m 27s–3m 48s)
    Runs
    50/50
    vs single+0.028not separablefew informative tasks2.6× the cost
  5. Rank 1, shared. Not separable from any other architecture.Orchestrator-workers0.9240.883–0.962
    Pass
    58% (39%–75%)
    Cost / run
    $0.216 ($0.155–$0.284)
    Latency
    55.7s (44.8–1m 7s)
    Runs
    50/50
    vs single+0.025not separablefew informative tasks1.8× the cost
  6. Rank 1, shared. Not separable from any other architecture.Single0.9000.843–0.947
    Pass
    54% (36%–71%)
    Cost / run
    $0.116 ($0.085–$0.150)
    Latency
    33.0s (24.7–41.8s)
    Runs
    50/50
    vs singlecontrol

Dot: mean score over tasks. Whisker: its interval. The thin vertical line is the single agent's mean. The axis is zoomed to 0.80–1.00. Runs: ranked of recorded, with the rest excluded and counted above.

Why these ranks

Each cell is the paired score difference, row minus column, over the tasks and repeats both ran. ▲ and ▼ mark the pairs the paired test separates, adjusted for the 15 comparisons made at once; ≈ marks a pair shown to lie within the equivalence margin. Everything else is a tie. A * says the data withheld an interval or a verdict from that pair: hover for why.

Paired score difference, row architecture minus column architecture, and whether the row scores reliably above, reliably below, equivalent within the margin, or is not separable from the column.
Row vs columnPropose-selectPlan-executePlan-execute-verifyGated-executeOrchestrator-workersSingle
Propose-select·+0.01 not separable; too few tasks carry the spread to show equivalence+0.01 not separable; too few tasks carry the spread to show equivalence+0.01 not separable; too few tasks carry the spread to show equivalence+0.01 not separable; too few tasks carry the spread to show equivalence+0.04 not separable; too few tasks carry the spread to show equivalence
Plan-execute−0.01 not separable; too few tasks carry the spread to show equivalence·0.00 not separable; too few tasks carry the spread to show equivalence+0.01 not separable+0.01 not separable; too few tasks carry the spread to show equivalence+0.03 not separable; too few tasks carry the spread to show equivalence
Plan-execute-verify−0.01 not separable; too few tasks carry the spread to show equivalence0.00 not separable; too few tasks carry the spread to show equivalence·0.00 not separable; too few tasks carry the spread to show equivalence+0.01 not separable; too few tasks carry the spread to show equivalence+0.03 not separable; too few tasks carry the spread to show equivalence
Gated-execute−0.01 not separable; too few tasks carry the spread to show equivalence−0.01 not separable0.00 not separable; too few tasks carry the spread to show equivalence·0.00 not separable; too few tasks carry the spread to show equivalence+0.03 not separable; too few tasks carry the spread to show equivalence
Orchestrator-workers−0.01 not separable; too few tasks carry the spread to show equivalence−0.01 not separable; too few tasks carry the spread to show equivalence−0.01 not separable; too few tasks carry the spread to show equivalence0.00 not separable; too few tasks carry the spread to show equivalence·+0.02 not separable; too few tasks carry the spread to show equivalence
Single−0.04 not separable; too few tasks carry the spread to show equivalence−0.03 not separable; too few tasks carry the spread to show equivalence−0.03 not separable; too few tasks carry the spread to show equivalence−0.03 not separable; too few tasks carry the spread to show equivalence−0.02 not separable; too few tasks carry the spread to show equivalence·

Does the structure pay for itself on Sonnet 5.5?

single, the control (1×, 0)equivalent to single (±0.05)Plan-executeOrchestrator-workersPlan-execute-verifyPropose-selectGated-execute
Sonnet 5.550 paired runs each−0.050+0.05+0.101×1.5×2×3×5×cost relative to single (log scale)Δ scorePlan-execute on Sonnet 5.5: +0.034 score [0.000, +0.087] at 1.3× the cost of single [1.3×, 1.4×], not separable from single (50 paired runs); too few tasks carry the spread to show equivalence.Orchestrator-workers on Sonnet 5.5: +0.025 score [−0.005, +0.078] at 1.8× the cost of single [1.5×, 2.2×], not separable from single (50 paired runs); too few tasks carry the spread to show equivalence.Plan-execute-verify on Sonnet 5.5: +0.031 score [−0.019, +0.077] at 2.4× the cost of single [2.2×, 2.6×], not separable from single (50 paired runs); too few tasks carry the spread to show equivalence.Propose-select on Sonnet 5.5: +0.040 score [+0.003, +0.095] at 4.1× the cost of single [3.8×, 4.3×], not separable from single (50 paired runs); too few tasks carry the spread to show equivalence.Gated-execute on Sonnet 5.5: +0.028 score [−0.012, +0.083] at 2.6× the cost of single [2.0×, 3.5×], not separable from single (50 paired runs); too few tasks carry the spread to show equivalence.singlePlan-executeOrchestrator-workersPlan-execute-verifyPropose-selectGated-execute
Vertical whiskers: interval on the score difference. Horizontal whiskers: interval on the cost multiple. A point without a whisker has no interval, and the table says why. Grey band: ±0.05, the equivalence margin. Every number is a mean over tasks, not runs: whole tasks are resampled (a cluster bootstrap of 10,000 replicates), so a task's repeats stay together and the 95% intervals are as wide as the tasks make them. Architectures are compared in pairs on the cells both ran, the same task and the same repeat: score differences are differences of means, cost and latency are geometric-mean ratios, and for them “better” means cheaper or faster. Each verdict comes from tests, not from whether two intervals overlap, and is Holm-adjusted within its family (one kind of contrast on one metric, across every model) to hold the chance of any false call in a family at 5%; so a verdict can read “not separable” while the unadjusted interval excludes zero. “Equivalent” means two one-sided tests show the difference within ±0.05 of score (×/÷1.1 for cost and latency). With fewer than 5 tasks no verdict is drawn, and where every task moved by the same amount there is no interval, which the table says instead of printing a zero-width one.
Show the numbers
ModelArchitectureΔ scoreintervalCost ×intervalVerdict
Sonnet 5.5Plan-execute+0.0340.000 to +0.0871.3×1.3×–1.4×inconclusive (too few tasks carry the spread to show equivalence)
Sonnet 5.5Orchestrator-workers+0.025−0.005 to +0.0781.8×1.5×–2.2×inconclusive (too few tasks carry the spread to show equivalence)
Sonnet 5.5Plan-execute-verify+0.031−0.019 to +0.0772.4×2.2×–2.6×inconclusive (too few tasks carry the spread to show equivalence)
Sonnet 5.5Propose-select+0.040+0.003 to +0.0954.1×3.8×–4.3×inconclusive (too few tasks carry the spread to show equivalence)
Sonnet 5.5Gated-execute+0.028−0.012 to +0.0832.6×2.0×–3.5×inconclusive (too few tasks carry the spread to show equivalence)
on the frontier: not shown to be beatenbeaten on one axis, matched on the other
0.800.850.900.951.00$0.080$0.150$0.200$0.300$0.500$0.800mean cost per run (USD, log scale)scoreSingle: score 0.900 [0.843, 0.947] at $0.116 a run [$0.085, $0.150], on the frontier.Plan-execute: score 0.933 [0.895, 0.966] at $0.146 a run [$0.113, $0.182], on the frontier.Orchestrator-workers: score 0.924 [0.883, 0.962] at $0.216 a run [$0.155, $0.284], on the frontier: behind another architecture on means, but the paired comparison does not separate them.Plan-execute-verify: score 0.930 [0.856, 0.985] at $0.278 a run [$0.207, $0.353], on the frontier: behind another architecture on means, but the paired comparison does not separate them.Propose-select: score 0.939 [0.903, 0.972] at $0.437 a run [$0.337, $0.538], on the frontier.Gated-execute: score 0.927 [0.889, 0.963] at $0.495 a run [$0.277, $0.736], on the frontier: behind another architecture on means, but the paired comparison does not separate them.SinglePlan-executeOrchestrator-workersPlan-execute-verifyPropose-selectGated-execute
0.800.850.900.951.00$0.080$0.150$0.400$0.800mean cost per run (USD, log scale)scoreSingle: score 0.900 [0.843, 0.947] at $0.116 a run [$0.085, $0.150], on the frontier.Plan-execute: score 0.933 [0.895, 0.966] at $0.146 a run [$0.113, $0.182], on the frontier.Orchestrator-workers: score 0.924 [0.883, 0.962] at $0.216 a run [$0.155, $0.284], on the frontier: behind another architecture on means, but the paired comparison does not separate them.Plan-execute-verify: score 0.930 [0.856, 0.985] at $0.278 a run [$0.207, $0.353], on the frontier: behind another architecture on means, but the paired comparison does not separate them.Propose-select: score 0.939 [0.903, 0.972] at $0.437 a run [$0.337, $0.538], on the frontier.Gated-execute: score 0.927 [0.889, 0.963] at $0.495 a run [$0.277, $0.736], on the frontier: behind another architecture on means, but the paired comparison does not separate them.SinglePlan-executeOrchestrator-workersPlan-execute-verifyPropose-selectGated-execute
Show the numbers
ArchitectureScoreintervalCost / runintervalFrontier
Single0.9000.843–0.947$0.116$0.085–$0.150frontier
Plan-execute0.9330.895–0.966$0.146$0.113–$0.182frontier
Orchestrator-workers0.9240.883–0.962$0.216$0.155–$0.284frontier
Plan-execute-verify0.9300.856–0.985$0.278$0.207–$0.353frontier
Propose-select0.9390.903–0.972$0.437$0.337–$0.538frontier
Gated-execute0.9270.889–0.963$0.495$0.277–$0.736frontier

Every number is a mean over tasks, not runs: whole tasks are resampled (a cluster bootstrap of 10,000 replicates), so a task's repeats stay together and the 95% intervals are as wide as the tasks make them. Architectures are compared in pairs on the cells both ran, the same task and the same repeat: score differences are differences of means, cost and latency are geometric-mean ratios, and for them “better” means cheaper or faster. Each verdict comes from tests, not from whether two intervals overlap, and is Holm-adjusted within its family (one kind of contrast on one metric, across every model) to hold the chance of any false call in a family at 5%; so a verdict can read “not separable” while the unadjusted interval excludes zero. “Equivalent” means two one-sided tests show the difference within ±0.05 of score (×/÷1.1 for cost and latency). With fewer than 5 tasks no verdict is drawn, and where every task moved by the same amount there is no interval, which the table says instead of printing a zero-width one. Computed over 300 ranked runs of 300 on this model.

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.