ScaffoldBench

Results · Claude Haiku 4.5

Every architecture on Haiku 4.5

Ranked by paired comparison: an architecture ranks below another only when, task by task and repeat by repeat, it scores reliably lower. Ties are shown as ties. Every number carries an interval bootstrapped over tasks, or says why it has none.

Runs
300
recorded on this model
Ranked
300
design verified, no API loss
Excluded: fidelity
0
the design did not run as designed
Excluded: API
0
lost to the API after retries
Retried
11
needed a second attempt

Harness health, check by check →

On Haiku 4.5, no architecture scores reliably above a single agent. 5 are not separable from it at this sample size.

Leaderboard

  1. Rank 1, shared. Not separable from any other architecture.Propose-select0.8090.711–0.901
    Pass
    52% (34%–70%)
    Cost / run
    $0.373 ($0.289–$0.460)
    Latency
    3m 31s (2m 39s–4m 30s)
    Runs
    50/50
    vs single+0.038not separable4.3× the cost
  2. Rank 1, shared. Not separable from any other architecture.Single0.7710.673–0.868
    Pass
    48% (30%–66%)
    Cost / run
    $0.091 ($0.068–$0.116)
    Latency
    1m 12s (52.1–1m 34s)
    Runs
    50/50
    vs singlecontrol
  3. Rank 1, shared. Not separable from any other architecture.Plan-execute0.7540.634–0.862
    Pass
    46% (28%–65%)
    Cost / run
    $0.128 ($0.103–$0.155)
    Latency
    1m 57s (1m 33s–2m 23s)
    Runs
    50/50
    vs single−0.017not separablefew informative tasks1.5× the cost
  4. Rank 1, shared. Not separable from any other architecture.Plan-execute-verify0.7230.588–0.845
    Pass
    44% (28%–62%)
    Cost / run
    $0.305 ($0.231–$0.386)
    Latency
    4m 39s (3m 28s–5m 58s)
    Runs
    50/50
    vs single−0.048not separablefew informative tasks3.3× the cost
  5. Rank 1, shared. Not separable from any other architecture.Orchestrator-workers0.7010.573–0.824
    Pass
    40% (24%–58%)
    Cost / run
    $0.171 ($0.134–$0.212)
    Latency
    1m 55s (1m 33s–2m 18s)
    Runs
    50/50
    vs single−0.070not separablefew informative tasks2.0× the cost
  6. Rank 1, shared. Not separable from any other architecture.Gated-execute0.6770.534–0.809
    Pass
    42% (26%–60%)
    Cost / run
    $0.862 ($0.474–$1.29)
    Latency
    12m 17s (6m 41s–18m 34s)
    Runs
    50/50
    vs single−0.094not separablefew informative tasks4.8× the cost

Dot: mean score over tasks. Whisker: its interval. The thin vertical line is the single agent's mean. The axis is zoomed to 0.50–1.00. Runs: ranked of recorded, with the rest excluded and counted above.

Why these ranks

Each cell is the paired score difference, row minus column, over the tasks and repeats both ran. ▲ and ▼ mark the pairs the paired test separates, adjusted for the 15 comparisons made at once; ≈ marks a pair shown to lie within the equivalence margin. Everything else is a tie. A * says the data withheld an interval or a verdict from that pair: hover for why.

Paired score difference, row architecture minus column architecture, and whether the row scores reliably above, reliably below, equivalent within the margin, or is not separable from the column.
Row vs columnPropose-selectSinglePlan-executePlan-execute-verifyOrchestrator-workersGated-execute
Propose-select·+0.04 not separable+0.06 not separable; too few tasks carry the spread to show equivalence+0.09 not separable; too few tasks carry the spread to show equivalence+0.11 not separable+0.13 not separable; too few tasks carry the spread to show equivalence
Single−0.04 not separable·+0.02 not separable; too few tasks carry the spread to show equivalence+0.05 not separable; too few tasks carry the spread to show equivalence+0.07 not separable; too few tasks carry the spread to show equivalence+0.09 not separable; too few tasks carry the spread to show equivalence
Plan-execute−0.06 not separable; too few tasks carry the spread to show equivalence−0.02 not separable; too few tasks carry the spread to show equivalence·+0.03 not separable; too few tasks carry the spread to show equivalence+0.05 not separable+0.08 not separable; too few tasks carry the spread to show equivalence
Plan-execute-verify−0.09 not separable; too few tasks carry the spread to show equivalence−0.05 not separable; too few tasks carry the spread to show equivalence−0.03 not separable; too few tasks carry the spread to show equivalence·+0.02 not separable; too few tasks carry the spread to show equivalence+0.05 not separable; too few tasks carry the spread to show equivalence
Orchestrator-workers−0.11 not separable−0.07 not separable; too few tasks carry the spread to show equivalence−0.05 not separable−0.02 not separable; too few tasks carry the spread to show equivalence·+0.02 not separable
Gated-execute−0.13 not separable; too few tasks carry the spread to show equivalence−0.09 not separable; too few tasks carry the spread to show equivalence−0.08 not separable; too few tasks carry the spread to show equivalence−0.05 not separable; too few tasks carry the spread to show equivalence−0.02 not separable·

Does the structure pay for itself on Haiku 4.5?

single, the control (1×, 0)equivalent to single (±0.05)Plan-executeOrchestrator-workersPlan-execute-verifyPropose-selectGated-execute
Haiku 4.550 paired runs each−0.20−0.100+0.101×1.5×3×5×8×cost relative to single (log scale)Δ scorePlan-execute on Haiku 4.5: −0.017 score [−0.143, +0.084] at 1.5× the cost of single [1.4×, 1.7×], not separable from single (50 paired runs); too few tasks carry the spread to show equivalence.Orchestrator-workers on Haiku 4.5: −0.070 score [−0.152, −0.001] at 2.0× the cost of single [1.6×, 2.5×], not separable from single (50 paired runs); too few tasks carry the spread to show equivalence.Plan-execute-verify on Haiku 4.5: −0.048 score [−0.171, +0.051] at 3.3× the cost of single [2.9×, 3.9×], not separable from single (50 paired runs); too few tasks carry the spread to show equivalence.Propose-select on Haiku 4.5: +0.038 score [−0.018, +0.094] at 4.3× the cost of single [4.0×, 4.7×], not separable from single (50 paired runs).Gated-execute on Haiku 4.5: −0.094 score [−0.215, +0.005] at 4.8× the cost of single [2.9×, 7.7×], not separable from single (50 paired runs); too few tasks carry the spread to show equivalence.singlePlan-executeOrchestrator-workersPlan-execute-verifyPropose-selectGated-execute
Vertical whiskers: interval on the score difference. Horizontal whiskers: interval on the cost multiple. A point without a whisker has no interval, and the table says why. Grey band: ±0.05, the equivalence margin. Every number is a mean over tasks, not runs: whole tasks are resampled (a cluster bootstrap of 10,000 replicates), so a task's repeats stay together and the 95% intervals are as wide as the tasks make them. Architectures are compared in pairs on the cells both ran, the same task and the same repeat: score differences are differences of means, cost and latency are geometric-mean ratios, and for them “better” means cheaper or faster. Each verdict comes from tests, not from whether two intervals overlap, and is Holm-adjusted within its family (one kind of contrast on one metric, across every model) to hold the chance of any false call in a family at 5%; so a verdict can read “not separable” while the unadjusted interval excludes zero. “Equivalent” means two one-sided tests show the difference within ±0.05 of score (×/÷1.1 for cost and latency). With fewer than 5 tasks no verdict is drawn, and where every task moved by the same amount there is no interval, which the table says instead of printing a zero-width one.
Show the numbers
ModelArchitectureΔ scoreintervalCost ×intervalVerdict
Haiku 4.5Plan-execute−0.017−0.143 to +0.0841.5×1.4×–1.7×inconclusive (too few tasks carry the spread to show equivalence)
Haiku 4.5Orchestrator-workers−0.070−0.152 to −0.0012.0×1.6×–2.5×inconclusive (too few tasks carry the spread to show equivalence)
Haiku 4.5Plan-execute-verify−0.048−0.171 to +0.0513.3×2.9×–3.9×inconclusive (too few tasks carry the spread to show equivalence)
Haiku 4.5Propose-select+0.038−0.018 to +0.0944.3×4.0×–4.7×inconclusive
Haiku 4.5Gated-execute−0.094−0.215 to +0.0054.8×2.9×–7.7×inconclusive (too few tasks carry the spread to show equivalence)
on the frontier: not shown to be beatenbeaten on one axis, matched on the other
0.500.600.700.800.901.00$0.060$0.100$0.200$0.400$0.800$1.50mean cost per run (USD, log scale)scoreSingle: score 0.771 [0.673, 0.868] at $0.091 a run [$0.068, $0.116], on the frontier.Plan-execute: score 0.754 [0.634, 0.862] at $0.128 a run [$0.103, $0.155], on the frontier: behind another architecture on means, but the paired comparison does not separate them.Orchestrator-workers: score 0.701 [0.573, 0.824] at $0.171 a run [$0.134, $0.212], on the frontier: behind another architecture on means, but the paired comparison does not separate them.Plan-execute-verify: score 0.723 [0.588, 0.845] at $0.305 a run [$0.231, $0.386], on the frontier: behind another architecture on means, but the paired comparison does not separate them.Propose-select: score 0.809 [0.711, 0.901] at $0.373 a run [$0.289, $0.460], on the frontier.Gated-execute: score 0.677 [0.534, 0.809] at $0.862 a run [$0.474, $1.29], on the frontier: behind another architecture on means, but the paired comparison does not separate them.SinglePlan-executeOrchestrator-workersPlan-execute-verifyPropose-selectGated-execute
0.500.600.700.800.901.00$0.060$0.200$0.500$1.50mean cost per run (USD, log scale)scoreSingle: score 0.771 [0.673, 0.868] at $0.091 a run [$0.068, $0.116], on the frontier.Plan-execute: score 0.754 [0.634, 0.862] at $0.128 a run [$0.103, $0.155], on the frontier: behind another architecture on means, but the paired comparison does not separate them.Orchestrator-workers: score 0.701 [0.573, 0.824] at $0.171 a run [$0.134, $0.212], on the frontier: behind another architecture on means, but the paired comparison does not separate them.Plan-execute-verify: score 0.723 [0.588, 0.845] at $0.305 a run [$0.231, $0.386], on the frontier: behind another architecture on means, but the paired comparison does not separate them.Propose-select: score 0.809 [0.711, 0.901] at $0.373 a run [$0.289, $0.460], on the frontier.Gated-execute: score 0.677 [0.534, 0.809] at $0.862 a run [$0.474, $1.29], on the frontier: behind another architecture on means, but the paired comparison does not separate them.SinglePlan-executeOrchestrator-workersPlan-execute-verifyPropose-selectGated-execute
Show the numbers
ArchitectureScoreintervalCost / runintervalFrontier
Single0.7710.673–0.868$0.091$0.068–$0.116frontier
Plan-execute0.7540.634–0.862$0.128$0.103–$0.155frontier
Orchestrator-workers0.7010.573–0.824$0.171$0.134–$0.212frontier
Plan-execute-verify0.7230.588–0.845$0.305$0.231–$0.386frontier
Propose-select0.8090.711–0.901$0.373$0.289–$0.460frontier
Gated-execute0.6770.534–0.809$0.862$0.474–$1.29frontier

Every number is a mean over tasks, not runs: whole tasks are resampled (a cluster bootstrap of 10,000 replicates), so a task's repeats stay together and the 95% intervals are as wide as the tasks make them. Architectures are compared in pairs on the cells both ran, the same task and the same repeat: score differences are differences of means, cost and latency are geometric-mean ratios, and for them “better” means cheaper or faster. Each verdict comes from tests, not from whether two intervals overlap, and is Holm-adjusted within its family (one kind of contrast on one metric, across every model) to hold the chance of any false call in a family at 5%; so a verdict can read “not separable” while the unadjusted interval excludes zero. “Equivalent” means two one-sided tests show the difference within ±0.05 of score (×/÷1.1 for cost and latency). With fewer than 5 tasks no verdict is drawn, and where every task moved by the same amount there is no interval, which the table says instead of printing a zero-width one. Computed over 300 ranked runs of 300 on this model.

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.