ScaffoldBench

Architecture

Propose-select

Does choosing among independent proposals beat committing to the first idea?

Three proposers investigate in parallel with read-only tools, a selector picks one plan, and an executor applies it.

How it runs

  1. 1The proposers

    read-only toolsruns in parallel

    Three proposers investigate at the same time with read-only tools, each starting from a different angle: the rules, the current state, or the outcome the person needs. Nothing is changed, and they never see one another.

  2. 2The selector

    no tools

    Reads the three proposals, picks the strongest and merges in what the others got right, through a structured reply. It holds no tools, so it chooses on the proposals alone.

  3. 3The executor

    full toolset

    Applies the chosen course of action with the full toolset. Until this call nothing in the world has changed.

Where it can fail quietly

A proposer that starts without its tools can only guess, and a selector that falls back to proposal 1 turns three investigations into one.

What the structure buys

Propose-select against a single agent on the same tasks and repeats, on each model. The interval is on the paired difference, so it is as wide as the evidence is thin.

  • Haiku 4.5=1 of 6
    Score
    0.809
    0.711–0.901
    Cost / run
    $0.373
    4.3× single
    Ranked
    50/50

    not separable from single+0.038 [−0.018, +0.094]

  • Sonnet 5.5=1 of 6
    Score
    0.939
    0.903–0.972
    Cost / run
    $0.437
    4.1× single
    Ranked
    50/50

    not separable from single+0.040 [+0.003, +0.095]too few tasks carry the spread to show equivalence

  • Opus 5.5=1 of 6
    Score
    0.958
    0.925–0.985
    Cost / run
    $0.928
    4.1× single
    Ranked
    50/50

    not separable from single+0.018 [0.000, +0.043]too few tasks carry the spread to show equivalence

Task by task against a single agent

A mean can hide whether a design helps everywhere a little or on one task a lot. Each bar is the difference from a single agent on one task; the best-served tasks are at the top.

Haiku 4.5
Sonnet 5.5
Opus 5.5
Haiku 4.5: level with a single agent on Escalate a payroll outage the requester wants closed
Sonnet 5.5: +0.50 against a single agent on Escalate a payroll outage the requester wants closed
Opus 5.5: level with a single agent on Escalate a payroll outage the requester wants closed
Haiku 4.5: −0.18 against a single agent on Audit a week of baggage fees
Sonnet 5.5: +0.30 against a single agent on Audit a week of baggage fees
Opus 5.5: +0.23 against a single agent on Audit a week of baggage fees
Haiku 4.5: +0.33 against a single agent on Work the access request queue
Sonnet 5.5: level with a single agent on Work the access request queue
Opus 5.5: level with a single agent on Work the access request queue
Haiku 4.5: +0.21 against a single agent on Work a cancellation queue where cash cannot be recalled
Sonnet 5.5: level with a single agent on Work a cancellation queue where cash cannot be recalled
Opus 5.5: +0.10 against a single agent on Work a cancellation queue where cash cannot be recalled
Haiku 4.5: +0.29 against a single agent on Work a morning's customer inbox
Sonnet 5.5: level with a single agent on Work a morning's customer inbox
Opus 5.5: level with a single agent on Work a morning's customer inbox
Haiku 4.5: +0.29 against a single agent on Add checked bags and collect the right fee
Sonnet 5.5: level with a single agent on Add checked bags and collect the right fee
Opus 5.5: level with a single agent on Add checked bags and collect the right fee
Haiku 4.5: +0.15 against a single agent on Share two short deliveries out among backorders
Sonnet 5.5: +0.09 against a single agent on Share two short deliveries out among backorders
Opus 5.5: level with a single agent on Share two short deliveries out among backorders
Haiku 4.5: +0.13 against a single agent on Triage the weekend's queue on Monday morning
Sonnet 5.5: level with a single agent on Triage the weekend's queue on Monday morning
Opus 5.5: level with a single agent on Triage the weekend's queue on Monday morning
Haiku 4.5: level with a single agent on NIST SP 800-63B password rules, current revision
Sonnet 5.5: level with a single agent on NIST SP 800-63B password rules, current revision
Opus 5.5: +0.13 against a single agent on NIST SP 800-63B password rules, current revision
Haiku 4.5: level with a single agent on Consolidate duplicate CRM reports
Sonnet 5.5: +0.08 against a single agent on Consolidate duplicate CRM reports
Opus 5.5: level with a single agent on Consolidate duplicate CRM reports
Haiku 4.5: +0.07 against a single agent on Run the major-incident runbook for a CI outage
Sonnet 5.5: level with a single agent on Run the major-incident runbook for a CI outage
Opus 5.5: level with a single agent on Run the major-incident runbook for a CI outage
Haiku 4.5: +0.06 against a single agent on Settle change requests under dated bulletins
Sonnet 5.5: level with a single agent on Settle change requests under dated bulletins
Opus 5.5: level with a single agent on Settle change requests under dated bulletins
Haiku 4.5: +0.06 against a single agent on Re-protect a cancelled flight when seats are scarce
Sonnet 5.5: level with a single agent on Re-protect a cancelled flight when seats are scarce
Opus 5.5: level with a single agent on Re-protect a cancelled flight when seats are scarce
Haiku 4.5: level with a single agent on Cancel a fare the customer wrongly believes is refundable to their card
Sonnet 5.5: level with a single agent on Cancel a fare the customer wrongly believes is refundable to their card
Opus 5.5: level with a single agent on Cancel a fare the customer wrongly believes is refundable to their card
Haiku 4.5: level with a single agent on Rebook onto the earliest available flight and settle the difference
Sonnet 5.5: level with a single agent on Rebook onto the earliest available flight and settle the difference
Opus 5.5: level with a single agent on Rebook onto the earliest available flight and settle the difference
Haiku 4.5: level with a single agent on Redirect the order that has not shipped
Sonnet 5.5: level with a single agent on Redirect the order that has not shipped
Opus 5.5: level with a single agent on Redirect the order that has not shipped
Haiku 4.5: level with a single agent on Return an expired, partially shipped order
Sonnet 5.5: level with a single agent on Return an expired, partially shipped order
Opus 5.5: level with a single agent on Return an expired, partially shipped order
Haiku 4.5: level with a single agent on Cancel one line of a multi-line order
Sonnet 5.5: level with a single agent on Cancel one line of a multi-line order
Opus 5.5: level with a single agent on Cancel one line of a multi-line order
Haiku 4.5: level with a single agent on Where ten HTTP header fields are defined
Sonnet 5.5: level with a single agent on Where ten HTTP header fields are defined
Opus 5.5: level with a single agent on Where ten HTTP header fields are defined
Haiku 4.5: level with a single agent on Twelve HTTP status codes, by the book
Sonnet 5.5: level with a single agent on Twelve HTTP status codes, by the book
Opus 5.5: level with a single agent on Twelve HTTP status codes, by the book
Haiku 4.5: level with a single agent on Triage a new service desk ticket
Sonnet 5.5: level with a single agent on Triage a new service desk ticket
Opus 5.5: level with a single agent on Triage a new service desk ticket
Haiku 4.5: level with a single agent on async vs defer on <script>
Sonnet 5.5: −0.03 against a single agent on async vs defer on <script>
Opus 5.5: level with a single agent on async vs defer on <script>
Haiku 4.5: −0.05 against a single agent on OWASP vs NIST on password hashing
Sonnet 5.5: level with a single agent on OWASP vs NIST on password hashing
Opus 5.5: level with a single agent on OWASP vs NIST on password hashing
Haiku 4.5: −0.09 against a single agent on Settle a desk of returns by what actually came back
Sonnet 5.5: level with a single agent on Settle a desk of returns by what actually came back
Opus 5.5: level with a single agent on Settle a desk of returns by what actually came back
Haiku 4.5: −0.31 against a single agent on HTTP 301 vs 308
Sonnet 5.5: +0.06 against a single agent on HTTP 301 vs 308
Opus 5.5: level with a single agent on HTTP 301 vs 308
Mean score of the architecture minus a single agent’s, over the repeats both completed, on a shared zero from −1 to +1. Runs that were excluded are left out.

Where the money goes

A run of Propose-select is 3 stages. This is what each one costs, so the price of the structure is separate from the price of the work.

  • Haiku 4.5$0.373 per run · 50 runs
  • Sonnet 5.5$0.437 per run · 50 runs
  • Opus 5.5$0.928 per run · 50 runs
StageHaiku 4.5Sonnet 5.5Opus 5.5
The proposers$0.24866%$0.29066%$0.61967%
The selector$0.05916%$0.05613%$0.11913%
The executor$0.06618%$0.09121%$0.19121%
Mean cost of a ranked run, by the stage of the design that spent it. Judge calls are not counted.

Did it run as designed?

Each check is recorded with every run. A run that fails one is excluded from every ranking and counted, never scored.

The design’s own checks

  • All three proposers ranpropose-select.proposals

    Three proposers ran in parallel, each from its own angle, and none was starved while another ran.

    150/150 100%
  • The selector chosepropose-select.selection

    The selector picked a proposal by a valid number, read from its schema reply and not a default.

    150/150 100%
  • Each call was shown the right toolspropose-select.tools

    Each proposer was shown the read-only tools, the selector none, and the executor the full toolset.

    150/150 100%
The harness’s checksall 10 held in every run
  • No API failuresharness.api

    No model call failed at the API.

    150/150 100%
  • Builtin toolsharness.builtin-tools

    A check recorded with the run.

    150/150 100%
  • Every call endedharness.calls-ended

    Every call ended with a result, or was stopped by the run's own limits.

    150/150 100%
  • No harness tool deniedharness.harness-tools

    The sandbox guard never denied a harness tool, nor a tool the session itself offered.

    150/150 100%
  • Judge answeredharness.judge

    At least one grading call returned a usable verdict.

    18/18 100%
  • Tool servers connectedharness.mcp-connected

    Every tool server the harness gave a call was connected when the session started.

    150/150 100%
  • No foreign tool serversharness.mcp-hermetic

    No call loaded a tool server the harness did not give it.

    150/150 100%
  • No silent fallbacksharness.not-degraded

    No phase fell back on a default (an empty plan, a default winner) instead of its output.

    150/150 100%
  • Run completedharness.run-completed

    The environment was built, the architecture returned and grading finished, none of them throwing.

    150/150 100%
  • Structured replies arrivedharness.structured-output

    Every call that asked for structured output and ran to the end got it.

    150/150 100%

Harness health across every architecture →

A run of it

Propose-select on Cancel a fare the customer wrongly believes is refundable to their card with Haiku 4.5, repeat 1. It passed.

5 calls · 2m 21s · $0.199

  • tool call
  • denied by the guard
  • tool returned an error
  • shown no tools (reasoning only)
  • lost its tools (server failed)
  • failed

The whole run, message by message →

150 ranked runs of 150 recorded.

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.