ScaffoldBench

Results · Claude Opus 5.5

Every architecture on Opus 5.5

Ranked by paired comparison: an architecture ranks below another only when, task by task and repeat by repeat, it scores reliably lower. Ties are shown as ties. Every number carries an interval bootstrapped over tasks, or says why it has none.

Runs
300
recorded on this model
Ranked
300
design verified, no API loss
Excluded: fidelity
0
the design did not run as designed
Excluded: API
0
lost to the API after retries
Retried
0
needed a second attempt

Harness health, check by check →

On Opus 5.5, no architecture scores reliably above a single agent. 5 are not separable from it at this sample size.

Leaderboard

  1. Rank 1, shared. Not separable from any other architecture.Propose-select0.9580.925–0.985
    Pass
    74% (55%–87%)
    Cost / run
    $0.928 ($0.716–$1.14)
    Latency
    1m 54s (1m 30s–2m 18s)
    Runs
    50/50
    vs single+0.018not separablefew informative tasks4.1× the cost
  2. Rank 1, shared. Not separable from any other architecture.Gated-execute0.9560.922–0.985
    Pass
    74% (55%–87%)
    Cost / run
    $0.934 ($0.535–$1.38)
    Latency
    3m 15s (1m 56s–4m 44s)
    Runs
    50/50
    vs single+0.016not separablefew informative tasks2.5× the cost
  3. Rank 1, shared. Not separable from any other architecture.Plan-execute-verify0.9470.890–0.986
    Pass
    76% (57%–88%)
    Cost / run
    $0.561 ($0.427–$0.696)
    Latency
    2m 2s (1m 34s–2m 31s)
    Runs
    50/50
    vs single+0.008not separablefew informative tasks2.4× the cost
  4. Rank 1, shared. Not separable from any other architecture.Plan-execute0.9470.913–0.977
    Pass
    66% (48%–80%)
    Cost / run
    $0.336 ($0.259–$0.414)
    Latency
    1m 19s (1m 2s–1m 36s)
    Runs
    50/50
    vs single+0.008not separablefew informative tasks1.4× the cost
  5. Rank 1, shared. Not separable from any other architecture.Single0.9390.903–0.974
    Pass
    66% (46%–81%)
    Cost / run
    $0.252 ($0.181–$0.325)
    Latency
    52.6s (39.0–1m 7s)
    Runs
    50/50
    vs singlecontrol
  6. Rank 1, shared. Not separable from any other architecture.Orchestrator-workers0.9210.869–0.961
    Pass
    60% (42%–76%)
    Cost / run
    $0.459 ($0.332–$0.596)
    Latency
    1m 26s (1m 9s–1m 45s)
    Runs
    50/50
    vs single−0.019not separablefew informative tasks1.8× the cost

Dot: mean score over tasks. Whisker: its interval. The thin vertical line is the single agent's mean. The axis is zoomed to 0.85–1.00. Runs: ranked of recorded, with the rest excluded and counted above.

Why these ranks

Each cell is the paired score difference, row minus column, over the tasks and repeats both ran. ▲ and ▼ mark the pairs the paired test separates, adjusted for the 15 comparisons made at once; ≈ marks a pair shown to lie within the equivalence margin. Everything else is a tie. A * says the data withheld an interval or a verdict from that pair: hover for why.

Paired score difference, row architecture minus column architecture, and whether the row scores reliably above, reliably below, equivalent within the margin, or is not separable from the column.
Row vs columnPropose-selectGated-executePlan-execute-verifyPlan-executeSingleOrchestrator-workers
Propose-select·0.00 not separable; too few tasks carry the spread to show equivalence+0.01 not separable; too few tasks carry the spread to show equivalence+0.01 not separable; too few tasks carry the spread to show equivalence+0.02 not separable; too few tasks carry the spread to show equivalence+0.04 not separable; too few tasks carry the spread to show equivalence
Gated-execute0.00 not separable; too few tasks carry the spread to show equivalence·+0.01 not separable+0.01 not separable; too few tasks carry the spread to show equivalence+0.02 not separable; too few tasks carry the spread to show equivalence+0.03 not separable; too few tasks carry the spread to show equivalence
Plan-execute-verify−0.01 not separable; too few tasks carry the spread to show equivalence−0.01 not separable·0.00 not separable; too few tasks carry the spread to show equivalence+0.01 not separable; too few tasks carry the spread to show equivalence+0.03 not separable; too few tasks carry the spread to show equivalence
Plan-execute−0.01 not separable; too few tasks carry the spread to show equivalence−0.01 not separable; too few tasks carry the spread to show equivalence0.00 not separable; too few tasks carry the spread to show equivalence·+0.01 not separable; too few tasks carry the spread to show equivalence+0.03 not separable; too few tasks carry the spread to show equivalence
Single−0.02 not separable; too few tasks carry the spread to show equivalence−0.02 not separable; too few tasks carry the spread to show equivalence−0.01 not separable; too few tasks carry the spread to show equivalence−0.01 not separable; too few tasks carry the spread to show equivalence·+0.02 not separable; too few tasks carry the spread to show equivalence
Orchestrator-workers−0.04 not separable; too few tasks carry the spread to show equivalence−0.03 not separable; too few tasks carry the spread to show equivalence−0.03 not separable; too few tasks carry the spread to show equivalence−0.03 not separable; too few tasks carry the spread to show equivalence−0.02 not separable; too few tasks carry the spread to show equivalence·

Does the structure pay for itself on Opus 5.5?

single, the control (1×, 0)equivalent to single (±0.05)Plan-executeOrchestrator-workersPlan-execute-verifyPropose-selectGated-execute
Opus 5.550 paired runs each−0.050+0.051×1.5×2×3×5×cost relative to single (log scale)Δ scorePlan-execute on Opus 5.5: +0.008 score [−0.020, +0.039] at 1.4× the cost of single [1.3×, 1.6×], not separable from single (50 paired runs); too few tasks carry the spread to show equivalence.Orchestrator-workers on Opus 5.5: −0.019 score [−0.051, +0.006] at 1.8× the cost of single [1.5×, 2.2×], not separable from single (50 paired runs); too few tasks carry the spread to show equivalence.Plan-execute-verify on Opus 5.5: +0.008 score [−0.042, +0.053] at 2.4× the cost of single [2.1×, 2.7×], not separable from single (50 paired runs); too few tasks carry the spread to show equivalence.Propose-select on Opus 5.5: +0.018 score [0.000, +0.043] at 4.1× the cost of single [3.7×, 4.5×], not separable from single (50 paired runs); too few tasks carry the spread to show equivalence.Gated-execute on Opus 5.5: +0.016 score [0.000, +0.041] at 2.5× the cost of single [1.9×, 3.2×], not separable from single (50 paired runs); too few tasks carry the spread to show equivalence.singlePlan-executeOrchestrator-workersPlan-execute-verifyPropose-selectGated-execute
Vertical whiskers: interval on the score difference. Horizontal whiskers: interval on the cost multiple. A point without a whisker has no interval, and the table says why. Grey band: ±0.05, the equivalence margin. Every number is a mean over tasks, not runs: whole tasks are resampled (a cluster bootstrap of 10,000 replicates), so a task's repeats stay together and the 95% intervals are as wide as the tasks make them. Architectures are compared in pairs on the cells both ran, the same task and the same repeat: score differences are differences of means, cost and latency are geometric-mean ratios, and for them “better” means cheaper or faster. Each verdict comes from tests, not from whether two intervals overlap, and is Holm-adjusted within its family (one kind of contrast on one metric, across every model) to hold the chance of any false call in a family at 5%; so a verdict can read “not separable” while the unadjusted interval excludes zero. “Equivalent” means two one-sided tests show the difference within ±0.05 of score (×/÷1.1 for cost and latency). With fewer than 5 tasks no verdict is drawn, and where every task moved by the same amount there is no interval, which the table says instead of printing a zero-width one.
Show the numbers
ModelArchitectureΔ scoreintervalCost ×intervalVerdict
Opus 5.5Plan-execute+0.008−0.020 to +0.0391.4×1.3×–1.6×inconclusive (too few tasks carry the spread to show equivalence)
Opus 5.5Orchestrator-workers−0.019−0.051 to +0.0061.8×1.5×–2.2×inconclusive (too few tasks carry the spread to show equivalence)
Opus 5.5Plan-execute-verify+0.008−0.042 to +0.0532.4×2.1×–2.7×inconclusive (too few tasks carry the spread to show equivalence)
Opus 5.5Propose-select+0.0180.000 to +0.0434.1×3.7×–4.5×inconclusive (too few tasks carry the spread to show equivalence)
Opus 5.5Gated-execute+0.0160.000 to +0.0412.5×1.9×–3.2×inconclusive (too few tasks carry the spread to show equivalence)
on the frontier: not shown to be beatenbeaten on one axis, matched on the other
0.850.900.951.00$0.200$0.300$0.500$0.600$1.00$1.50mean cost per run (USD, log scale)scoreSingle: score 0.939 [0.903, 0.974] at $0.252 a run [$0.181, $0.325], on the frontier.Plan-execute: score 0.947 [0.913, 0.977] at $0.336 a run [$0.259, $0.414], on the frontier.Orchestrator-workers: score 0.921 [0.869, 0.961] at $0.459 a run [$0.332, $0.596], on the frontier: behind another architecture on means, but the paired comparison does not separate them.Plan-execute-verify: score 0.947 [0.890, 0.986] at $0.561 a run [$0.427, $0.696], on the frontier.Propose-select: score 0.958 [0.925, 0.985] at $0.928 a run [$0.716, $1.14], on the frontier.Gated-execute: score 0.956 [0.922, 0.985] at $0.934 a run [$0.535, $1.38], on the frontier: behind another architecture on means, but the paired comparison does not separate them.SinglePlan-executeOrchestrator-workersPlan-execute-verifyPropose-selectGated-execute
0.850.900.951.00$0.200$0.400$0.800$1.50mean cost per run (USD, log scale)scoreSingle: score 0.939 [0.903, 0.974] at $0.252 a run [$0.181, $0.325], on the frontier.Plan-execute: score 0.947 [0.913, 0.977] at $0.336 a run [$0.259, $0.414], on the frontier.Orchestrator-workers: score 0.921 [0.869, 0.961] at $0.459 a run [$0.332, $0.596], on the frontier: behind another architecture on means, but the paired comparison does not separate them.Plan-execute-verify: score 0.947 [0.890, 0.986] at $0.561 a run [$0.427, $0.696], on the frontier.Propose-select: score 0.958 [0.925, 0.985] at $0.928 a run [$0.716, $1.14], on the frontier.Gated-execute: score 0.956 [0.922, 0.985] at $0.934 a run [$0.535, $1.38], on the frontier: behind another architecture on means, but the paired comparison does not separate them.SinglePlan-executeOrchestrator-workersPlan-execute-verifyPropose-selectGated-execute
Show the numbers
ArchitectureScoreintervalCost / runintervalFrontier
Single0.9390.903–0.974$0.252$0.181–$0.325frontier
Plan-execute0.9470.913–0.977$0.336$0.259–$0.414frontier
Orchestrator-workers0.9210.869–0.961$0.459$0.332–$0.596frontier
Plan-execute-verify0.9470.890–0.986$0.561$0.427–$0.696frontier
Propose-select0.9580.925–0.985$0.928$0.716–$1.14frontier
Gated-execute0.9560.922–0.985$0.934$0.535–$1.38frontier

Every number is a mean over tasks, not runs: whole tasks are resampled (a cluster bootstrap of 10,000 replicates), so a task's repeats stay together and the 95% intervals are as wide as the tasks make them. Architectures are compared in pairs on the cells both ran, the same task and the same repeat: score differences are differences of means, cost and latency are geometric-mean ratios, and for them “better” means cheaper or faster. Each verdict comes from tests, not from whether two intervals overlap, and is Holm-adjusted within its family (one kind of contrast on one metric, across every model) to hold the chance of any false call in a family at 5%; so a verdict can read “not separable” while the unadjusted interval excludes zero. “Equivalent” means two one-sided tests show the difference within ±0.05 of score (×/÷1.1 for cost and latency). With fewer than 5 tasks no verdict is drawn, and where every task moved by the same amount there is no interval, which the table says instead of printing a zero-width one. Computed over 300 ranked runs of 300 on this model.

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.