ScaffoldBench

Score against cost

Which architecture should I use?

Every architecture on the plane it is actually chosen on. A point leaves the frontier only when another is shown, by the paired comparison, to beat it on score or on cost and to match it on the other. The whiskers are the intervals on each axis; where nothing separates two points, both stay.

Haiku 4.5

On Haiku 4.5, the frontier runs through Single, Plan-execute, Orchestrator-workers, Plan-execute-verify, Propose-select and Gated-execute, starting at $0.091 a run. Plan-execute, Orchestrator-workers, Plan-execute-verify and Gated-execute trail another point on means, but the paired comparison does not separate them, so they stay. No architecture scores reliably above a single agent here, so these intervals support no extra spend.

on the frontier: not shown to be beatenbeaten on one axis, matched on the other
0.500.600.700.800.901.00$0.060$0.100$0.200$0.400$0.800$1.50mean cost per run (USD, log scale)scoreSingle: score 0.771 [0.673, 0.868] at $0.091 a run [$0.068, $0.116], on the frontier.Plan-execute: score 0.754 [0.634, 0.862] at $0.128 a run [$0.103, $0.155], on the frontier: behind another architecture on means, but the paired comparison does not separate them.Orchestrator-workers: score 0.701 [0.573, 0.824] at $0.171 a run [$0.134, $0.212], on the frontier: behind another architecture on means, but the paired comparison does not separate them.Plan-execute-verify: score 0.723 [0.588, 0.845] at $0.305 a run [$0.231, $0.386], on the frontier: behind another architecture on means, but the paired comparison does not separate them.Propose-select: score 0.809 [0.711, 0.901] at $0.373 a run [$0.289, $0.460], on the frontier.Gated-execute: score 0.677 [0.534, 0.809] at $0.862 a run [$0.474, $1.29], on the frontier: behind another architecture on means, but the paired comparison does not separate them.SinglePlan-executeOrchestrator-workersPlan-execute-verifyPropose-selectGated-execute
0.500.600.700.800.901.00$0.060$0.200$0.500$1.50mean cost per run (USD, log scale)scoreSingle: score 0.771 [0.673, 0.868] at $0.091 a run [$0.068, $0.116], on the frontier.Plan-execute: score 0.754 [0.634, 0.862] at $0.128 a run [$0.103, $0.155], on the frontier: behind another architecture on means, but the paired comparison does not separate them.Orchestrator-workers: score 0.701 [0.573, 0.824] at $0.171 a run [$0.134, $0.212], on the frontier: behind another architecture on means, but the paired comparison does not separate them.Plan-execute-verify: score 0.723 [0.588, 0.845] at $0.305 a run [$0.231, $0.386], on the frontier: behind another architecture on means, but the paired comparison does not separate them.Propose-select: score 0.809 [0.711, 0.901] at $0.373 a run [$0.289, $0.460], on the frontier.Gated-execute: score 0.677 [0.534, 0.809] at $0.862 a run [$0.474, $1.29], on the frontier: behind another architecture on means, but the paired comparison does not separate them.SinglePlan-executeOrchestrator-workersPlan-execute-verifyPropose-selectGated-execute
Show the numbers
ArchitectureScoreintervalCost / runintervalFrontier
Single0.7710.673–0.868$0.091$0.068–$0.116frontier
Plan-execute0.7540.634–0.862$0.128$0.103–$0.155frontier
Orchestrator-workers0.7010.573–0.824$0.171$0.134–$0.212frontier
Plan-execute-verify0.7230.588–0.845$0.305$0.231–$0.386frontier
Propose-select0.8090.711–0.901$0.373$0.289–$0.460frontier
Gated-execute0.6770.534–0.809$0.862$0.474–$1.29frontier

Sonnet 5.5

On Sonnet 5.5, the frontier runs through Single, Plan-execute, Orchestrator-workers, Plan-execute-verify, Propose-select and Gated-execute, starting at $0.116 a run. Orchestrator-workers, Plan-execute-verify and Gated-execute trail another point on means, but the paired comparison does not separate them, so they stay. No architecture scores reliably above a single agent here, so these intervals support no extra spend.

on the frontier: not shown to be beatenbeaten on one axis, matched on the other
0.800.850.900.951.00$0.080$0.150$0.200$0.300$0.500$0.800mean cost per run (USD, log scale)scoreSingle: score 0.900 [0.843, 0.947] at $0.116 a run [$0.085, $0.150], on the frontier.Plan-execute: score 0.933 [0.895, 0.966] at $0.146 a run [$0.113, $0.182], on the frontier.Orchestrator-workers: score 0.924 [0.883, 0.962] at $0.216 a run [$0.155, $0.284], on the frontier: behind another architecture on means, but the paired comparison does not separate them.Plan-execute-verify: score 0.930 [0.856, 0.985] at $0.278 a run [$0.207, $0.353], on the frontier: behind another architecture on means, but the paired comparison does not separate them.Propose-select: score 0.939 [0.903, 0.972] at $0.437 a run [$0.337, $0.538], on the frontier.Gated-execute: score 0.927 [0.889, 0.963] at $0.495 a run [$0.277, $0.736], on the frontier: behind another architecture on means, but the paired comparison does not separate them.SinglePlan-executeOrchestrator-workersPlan-execute-verifyPropose-selectGated-execute
0.800.850.900.951.00$0.080$0.150$0.400$0.800mean cost per run (USD, log scale)scoreSingle: score 0.900 [0.843, 0.947] at $0.116 a run [$0.085, $0.150], on the frontier.Plan-execute: score 0.933 [0.895, 0.966] at $0.146 a run [$0.113, $0.182], on the frontier.Orchestrator-workers: score 0.924 [0.883, 0.962] at $0.216 a run [$0.155, $0.284], on the frontier: behind another architecture on means, but the paired comparison does not separate them.Plan-execute-verify: score 0.930 [0.856, 0.985] at $0.278 a run [$0.207, $0.353], on the frontier: behind another architecture on means, but the paired comparison does not separate them.Propose-select: score 0.939 [0.903, 0.972] at $0.437 a run [$0.337, $0.538], on the frontier.Gated-execute: score 0.927 [0.889, 0.963] at $0.495 a run [$0.277, $0.736], on the frontier: behind another architecture on means, but the paired comparison does not separate them.SinglePlan-executeOrchestrator-workersPlan-execute-verifyPropose-selectGated-execute
Show the numbers
ArchitectureScoreintervalCost / runintervalFrontier
Single0.9000.843–0.947$0.116$0.085–$0.150frontier
Plan-execute0.9330.895–0.966$0.146$0.113–$0.182frontier
Orchestrator-workers0.9240.883–0.962$0.216$0.155–$0.284frontier
Plan-execute-verify0.9300.856–0.985$0.278$0.207–$0.353frontier
Propose-select0.9390.903–0.972$0.437$0.337–$0.538frontier
Gated-execute0.9270.889–0.963$0.495$0.277–$0.736frontier

Opus 5.5

On Opus 5.5, the frontier runs through Single, Plan-execute, Orchestrator-workers, Plan-execute-verify, Propose-select and Gated-execute, starting at $0.252 a run. Orchestrator-workers and Gated-execute trail another point on means, but the paired comparison does not separate them, so they stay. No architecture scores reliably above a single agent here, so these intervals support no extra spend.

on the frontier: not shown to be beatenbeaten on one axis, matched on the other
0.850.900.951.00$0.200$0.300$0.500$0.600$1.00$1.50mean cost per run (USD, log scale)scoreSingle: score 0.939 [0.903, 0.974] at $0.252 a run [$0.181, $0.325], on the frontier.Plan-execute: score 0.947 [0.913, 0.977] at $0.336 a run [$0.259, $0.414], on the frontier.Orchestrator-workers: score 0.921 [0.869, 0.961] at $0.459 a run [$0.332, $0.596], on the frontier: behind another architecture on means, but the paired comparison does not separate them.Plan-execute-verify: score 0.947 [0.890, 0.986] at $0.561 a run [$0.427, $0.696], on the frontier.Propose-select: score 0.958 [0.925, 0.985] at $0.928 a run [$0.716, $1.14], on the frontier.Gated-execute: score 0.956 [0.922, 0.985] at $0.934 a run [$0.535, $1.38], on the frontier: behind another architecture on means, but the paired comparison does not separate them.SinglePlan-executeOrchestrator-workersPlan-execute-verifyPropose-selectGated-execute
0.850.900.951.00$0.200$0.400$0.800$1.50mean cost per run (USD, log scale)scoreSingle: score 0.939 [0.903, 0.974] at $0.252 a run [$0.181, $0.325], on the frontier.Plan-execute: score 0.947 [0.913, 0.977] at $0.336 a run [$0.259, $0.414], on the frontier.Orchestrator-workers: score 0.921 [0.869, 0.961] at $0.459 a run [$0.332, $0.596], on the frontier: behind another architecture on means, but the paired comparison does not separate them.Plan-execute-verify: score 0.947 [0.890, 0.986] at $0.561 a run [$0.427, $0.696], on the frontier.Propose-select: score 0.958 [0.925, 0.985] at $0.928 a run [$0.716, $1.14], on the frontier.Gated-execute: score 0.956 [0.922, 0.985] at $0.934 a run [$0.535, $1.38], on the frontier: behind another architecture on means, but the paired comparison does not separate them.SinglePlan-executeOrchestrator-workersPlan-execute-verifyPropose-selectGated-execute
Show the numbers
ArchitectureScoreintervalCost / runintervalFrontier
Single0.9390.903–0.974$0.252$0.181–$0.325frontier
Plan-execute0.9470.913–0.977$0.336$0.259–$0.414frontier
Orchestrator-workers0.9210.869–0.961$0.459$0.332–$0.596frontier
Plan-execute-verify0.9470.890–0.986$0.561$0.427–$0.696frontier
Propose-select0.9580.925–0.985$0.928$0.716–$1.14frontier
Gated-execute0.9560.922–0.985$0.934$0.535–$1.38frontier

Every number is a mean over tasks, not runs: whole tasks are resampled (a cluster bootstrap of 10,000 replicates), so a task's repeats stay together and the 95% intervals are as wide as the tasks make them. Architectures are compared in pairs on the cells both ran, the same task and the same repeat: score differences are differences of means, cost and latency are geometric-mean ratios, and for them “better” means cheaper or faster. Each verdict comes from tests, not from whether two intervals overlap, and is Holm-adjusted within its family (one kind of contrast on one metric, across every model) to hold the chance of any false call in a family at 5%; so a verdict can read “not separable” while the unadjusted interval excludes zero. “Equivalent” means two one-sided tests show the difference within ±0.05 of score (×/÷1.1 for cost and latency). With fewer than 5 tasks no verdict is drawn, and where every task moved by the same amount there is no interval, which the table says instead of printing a zero-width one.

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.