Score against cost
Which architecture should I use?
Every architecture on the plane it is actually chosen on. A point leaves the frontier only when another is shown, by the paired comparison, to beat it on score or on cost and to match it on the other. The whiskers are the intervals on each axis; where nothing separates two points, both stay.
Haiku 4.5
On Haiku 4.5, the frontier runs through Single, Plan-execute, Orchestrator-workers, Plan-execute-verify, Propose-select and Gated-execute, starting at $0.091 a run. Plan-execute, Orchestrator-workers, Plan-execute-verify and Gated-execute trail another point on means, but the paired comparison does not separate them, so they stay. No architecture scores reliably above a single agent here, so these intervals support no extra spend.
Show the numbers
| Architecture | Score | interval | Cost / run | interval | Frontier |
|---|---|---|---|---|---|
| Single | 0.771 | 0.673–0.868 | $0.091 | $0.068–$0.116 | frontier |
| Plan-execute | 0.754 | 0.634–0.862 | $0.128 | $0.103–$0.155 | frontier |
| Orchestrator-workers | 0.701 | 0.573–0.824 | $0.171 | $0.134–$0.212 | frontier |
| Plan-execute-verify | 0.723 | 0.588–0.845 | $0.305 | $0.231–$0.386 | frontier |
| Propose-select | 0.809 | 0.711–0.901 | $0.373 | $0.289–$0.460 | frontier |
| Gated-execute | 0.677 | 0.534–0.809 | $0.862 | $0.474–$1.29 | frontier |
Sonnet 5.5
On Sonnet 5.5, the frontier runs through Single, Plan-execute, Orchestrator-workers, Plan-execute-verify, Propose-select and Gated-execute, starting at $0.116 a run. Orchestrator-workers, Plan-execute-verify and Gated-execute trail another point on means, but the paired comparison does not separate them, so they stay. No architecture scores reliably above a single agent here, so these intervals support no extra spend.
Show the numbers
| Architecture | Score | interval | Cost / run | interval | Frontier |
|---|---|---|---|---|---|
| Single | 0.900 | 0.843–0.947 | $0.116 | $0.085–$0.150 | frontier |
| Plan-execute | 0.933 | 0.895–0.966 | $0.146 | $0.113–$0.182 | frontier |
| Orchestrator-workers | 0.924 | 0.883–0.962 | $0.216 | $0.155–$0.284 | frontier |
| Plan-execute-verify | 0.930 | 0.856–0.985 | $0.278 | $0.207–$0.353 | frontier |
| Propose-select | 0.939 | 0.903–0.972 | $0.437 | $0.337–$0.538 | frontier |
| Gated-execute | 0.927 | 0.889–0.963 | $0.495 | $0.277–$0.736 | frontier |
Opus 5.5
On Opus 5.5, the frontier runs through Single, Plan-execute, Orchestrator-workers, Plan-execute-verify, Propose-select and Gated-execute, starting at $0.252 a run. Orchestrator-workers and Gated-execute trail another point on means, but the paired comparison does not separate them, so they stay. No architecture scores reliably above a single agent here, so these intervals support no extra spend.
Show the numbers
| Architecture | Score | interval | Cost / run | interval | Frontier |
|---|---|---|---|---|---|
| Single | 0.939 | 0.903–0.974 | $0.252 | $0.181–$0.325 | frontier |
| Plan-execute | 0.947 | 0.913–0.977 | $0.336 | $0.259–$0.414 | frontier |
| Orchestrator-workers | 0.921 | 0.869–0.961 | $0.459 | $0.332–$0.596 | frontier |
| Plan-execute-verify | 0.947 | 0.890–0.986 | $0.561 | $0.427–$0.696 | frontier |
| Propose-select | 0.958 | 0.925–0.985 | $0.928 | $0.716–$1.14 | frontier |
| Gated-execute | 0.956 | 0.922–0.985 | $0.934 | $0.535–$1.38 | frontier |
Every number is a mean over tasks, not runs: whole tasks are resampled (a cluster bootstrap of 10,000 replicates), so a task's repeats stay together and the 95% intervals are as wide as the tasks make them. Architectures are compared in pairs on the cells both ran, the same task and the same repeat: score differences are differences of means, cost and latency are geometric-mean ratios, and for them “better” means cheaper or faster. Each verdict comes from tests, not from whether two intervals overlap, and is Holm-adjusted within its family (one kind of contrast on one metric, across every model) to hold the chance of any false call in a family at 5%; so a verdict can read “not separable” while the unadjusted interval excludes zero. “Equivalent” means two one-sided tests show the difference within ±0.05 of score (×/÷1.1 for cost and latency). With fewer than 5 tasks no verdict is drawn, and where every task moved by the same amount there is no interval, which the table says instead of printing a zero-width one.
Provenance of the numbers on this page
- run set
- scaffold-or-scale-v1
- models
- Haiku 4.5, Sonnet 5.5, Opus 5.5
- effort
- high
- repeats
- 2
- fixtures
- replay
- runs
- 900 / 900
- strategy cost
- $354.44
- judge cost
- $9.19
- wall clock
- 9h 48m
- git
- fce5807b2d
Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.