ScaffoldBench

Architecture

Plan-execute-verify

Is a second opinion on the finished state worth the cost of asking for it?

Plan and execute, then a verifier with read-only tools checks the real state and can send the work back.

How it runs

  1. 1The planner

    no tools

    The same tool-less planner as in plan-execute: an ordered list of steps through a structured reply.

  2. 2The executor

    full toolset

    Carries out the plan with the full toolset. When the verifier sends the work back, a revision call, with the full toolset again, acts on its issues, at most twice.

  3. 3The verifier

    read-only tools

    Holds only the read-only tools and has to check the real state of the world rather than the executor’s account of it. It returns a verdict with a severity: a blocking issue sends the work back, a minor one ends the review.

Where it can fail quietly

A verifier whose verdict cannot be read, or that approves by default, never sends anything back: the loop is dead code and the cost is overhead.

What the structure buys

Plan-execute-verify against a single agent on the same tasks and repeats, on each model. The interval is on the paired difference, so it is as wide as the evidence is thin.

  • Haiku 4.5=1 of 6
    Score
    0.723
    0.588–0.845
    Cost / run
    $0.305
    3.3× single
    Ranked
    50/50

    not separable from single−0.048 [−0.171, +0.051]too few tasks carry the spread to show equivalence

  • Sonnet 5.5=1 of 6
    Score
    0.930
    0.856–0.985
    Cost / run
    $0.278
    2.4× single
    Ranked
    50/50

    not separable from single+0.031 [−0.019, +0.077]too few tasks carry the spread to show equivalence

  • Opus 5.5=1 of 6
    Score
    0.947
    0.890–0.986
    Cost / run
    $0.561
    2.4× single
    Ranked
    50/50

    not separable from single+0.008 [−0.042, +0.053]too few tasks carry the spread to show equivalence

Task by task against a single agent

A mean can hide whether a design helps everywhere a little or on one task a lot. Each bar is the difference from a single agent on one task; the best-served tasks are at the top.

Haiku 4.5
Sonnet 5.5
Opus 5.5
Haiku 4.5: +0.27 against a single agent on Work a morning's customer inbox
Sonnet 5.5: +0.24 against a single agent on Work a morning's customer inbox
Opus 5.5: +0.24 against a single agent on Work a morning's customer inbox
Haiku 4.5: +0.36 against a single agent on Triage the weekend's queue on Monday morning
Sonnet 5.5: +0.20 against a single agent on Triage the weekend's queue on Monday morning
Opus 5.5: +0.12 against a single agent on Triage the weekend's queue on Monday morning
Haiku 4.5: +0.09 against a single agent on Work a cancellation queue where cash cannot be recalled
Sonnet 5.5: +0.20 against a single agent on Work a cancellation queue where cash cannot be recalled
Opus 5.5: +0.20 against a single agent on Work a cancellation queue where cash cannot be recalled
Haiku 4.5: +0.04 against a single agent on Settle a desk of returns by what actually came back
Sonnet 5.5: +0.15 against a single agent on Settle a desk of returns by what actually came back
Opus 5.5: +0.15 against a single agent on Settle a desk of returns by what actually came back
Haiku 4.5: +0.29 against a single agent on Add checked bags and collect the right fee
Sonnet 5.5: level with a single agent on Add checked bags and collect the right fee
Opus 5.5: level with a single agent on Add checked bags and collect the right fee
Haiku 4.5: +0.13 against a single agent on Settle change requests under dated bulletins
Sonnet 5.5: level with a single agent on Settle change requests under dated bulletins
Opus 5.5: level with a single agent on Settle change requests under dated bulletins
Haiku 4.5: level with a single agent on Consolidate duplicate CRM reports
Sonnet 5.5: +0.08 against a single agent on Consolidate duplicate CRM reports
Opus 5.5: level with a single agent on Consolidate duplicate CRM reports
Haiku 4.5: −0.18 against a single agent on Share two short deliveries out among backorders
Sonnet 5.5: +0.19 against a single agent on Share two short deliveries out among backorders
Opus 5.5: level with a single agent on Share two short deliveries out among backorders
Haiku 4.5: level with a single agent on Cancel a fare the customer wrongly believes is refundable to their card
Sonnet 5.5: level with a single agent on Cancel a fare the customer wrongly believes is refundable to their card
Opus 5.5: level with a single agent on Cancel a fare the customer wrongly believes is refundable to their card
Haiku 4.5: level with a single agent on Redirect the order that has not shipped
Sonnet 5.5: level with a single agent on Redirect the order that has not shipped
Opus 5.5: level with a single agent on Redirect the order that has not shipped
Haiku 4.5: level with a single agent on Return an expired, partially shipped order
Sonnet 5.5: level with a single agent on Return an expired, partially shipped order
Opus 5.5: level with a single agent on Return an expired, partially shipped order
Haiku 4.5: level with a single agent on Cancel one line of a multi-line order
Sonnet 5.5: level with a single agent on Cancel one line of a multi-line order
Opus 5.5: level with a single agent on Cancel one line of a multi-line order
Haiku 4.5: level with a single agent on HTTP 301 vs 308
Sonnet 5.5: level with a single agent on HTTP 301 vs 308
Opus 5.5: level with a single agent on HTTP 301 vs 308
Haiku 4.5: level with a single agent on Twelve HTTP status codes, by the book
Sonnet 5.5: level with a single agent on Twelve HTTP status codes, by the book
Opus 5.5: level with a single agent on Twelve HTTP status codes, by the book
Haiku 4.5: level with a single agent on Triage a new service desk ticket
Sonnet 5.5: level with a single agent on Triage a new service desk ticket
Opus 5.5: level with a single agent on Triage a new service desk ticket
Haiku 4.5: −0.21 against a single agent on Work the access request queue
Sonnet 5.5: +0.09 against a single agent on Work the access request queue
Opus 5.5: +0.09 against a single agent on Work the access request queue
Haiku 4.5: level with a single agent on NIST SP 800-63B password rules, current revision
Sonnet 5.5: level with a single agent on NIST SP 800-63B password rules, current revision
Opus 5.5: −0.08 against a single agent on NIST SP 800-63B password rules, current revision
Haiku 4.5: +0.01 against a single agent on OWASP vs NIST on password hashing
Sonnet 5.5: level with a single agent on OWASP vs NIST on password hashing
Opus 5.5: −0.09 against a single agent on OWASP vs NIST on password hashing
Haiku 4.5: −0.05 against a single agent on async vs defer on <script>
Sonnet 5.5: level with a single agent on async vs defer on <script>
Opus 5.5: −0.05 against a single agent on async vs defer on <script>
Haiku 4.5: level with a single agent on Where ten HTTP header fields are defined
Sonnet 5.5: level with a single agent on Where ten HTTP header fields are defined
Opus 5.5: −0.13 against a single agent on Where ten HTTP header fields are defined
Haiku 4.5: +0.05 against a single agent on Run the major-incident runbook for a CI outage
Sonnet 5.5: +0.09 against a single agent on Run the major-incident runbook for a CI outage
Opus 5.5: −0.36 against a single agent on Run the major-incident runbook for a CI outage
Haiku 4.5: −0.12 against a single agent on Audit a week of baggage fees
Sonnet 5.5: −0.34 against a single agent on Audit a week of baggage fees
Opus 5.5: +0.11 against a single agent on Audit a week of baggage fees
Haiku 4.5: −0.37 against a single agent on Re-protect a cancelled flight when seats are scarce
Sonnet 5.5: −0.12 against a single agent on Re-protect a cancelled flight when seats are scarce
Opus 5.5: level with a single agent on Re-protect a cancelled flight when seats are scarce
Haiku 4.5: −0.50 against a single agent on Rebook onto the earliest available flight and settle the difference
Sonnet 5.5: level with a single agent on Rebook onto the earliest available flight and settle the difference
Opus 5.5: level with a single agent on Rebook onto the earliest available flight and settle the difference
Haiku 4.5: −1.00 against a single agent on Escalate a payroll outage the requester wants closed
Sonnet 5.5: level with a single agent on Escalate a payroll outage the requester wants closed
Opus 5.5: level with a single agent on Escalate a payroll outage the requester wants closed
Mean score of the architecture minus a single agent’s, over the repeats both completed, on a shared zero from −1 to +1. Runs that were excluded are left out.

Where the money goes

A run of Plan-execute-verify is 3 stages. This is what each one costs, so the price of the structure is separate from the price of the work.

  • Haiku 4.5$0.305 per run · 50 runs
  • Sonnet 5.5$0.278 per run · 50 runs
  • Opus 5.5$0.561 per run · 50 runs
StageHaiku 4.5Sonnet 5.5Opus 5.5
The planner$0.0258%$0.0155%$0.0346%
The executor$0.14146%$0.15656%$0.31757%
The verifier$0.14046%$0.10638%$0.20937%
Mean cost of a ranked run, by the stage of the design that spent it. Judge calls are not counted.

Did it run as designed?

Each check is recorded with every run. A run that fails one is excluded from every ranking and counted, never scored.

The design’s own checks

  • The plan was a planplan-execute-verify.plan

    The planner’s reply was read through its schema and held at least one step.

    150/150 100%
  • Every verdict was readplan-execute-verify.verdicts

    Every review round ended with a verdict read from the schema reply, not a default.

    150/150 100%
  • The loop did what its verdicts saidplan-execute-verify.loop

    Every verdict that called for a revision got one, and nothing was revised without such a verdict.

    150/150 100%
  • Each call was shown the right toolsplan-execute-verify.tools

    The planner was shown no tools, the verifier the read-only ones, and the executor and reviser the full toolset.

    150/150 100%
The harness’s checksall 10 held in every run
  • No API failuresharness.api

    No model call failed at the API.

    150/150 100%
  • Builtin toolsharness.builtin-tools

    A check recorded with the run.

    150/150 100%
  • Every call endedharness.calls-ended

    Every call ended with a result, or was stopped by the run's own limits.

    150/150 100%
  • No harness tool deniedharness.harness-tools

    The sandbox guard never denied a harness tool, nor a tool the session itself offered.

    150/150 100%
  • Judge answeredharness.judge

    At least one grading call returned a usable verdict.

    18/18 100%
  • Tool servers connectedharness.mcp-connected

    Every tool server the harness gave a call was connected when the session started.

    150/150 100%
  • No foreign tool serversharness.mcp-hermetic

    No call loaded a tool server the harness did not give it.

    150/150 100%
  • No silent fallbacksharness.not-degraded

    No phase fell back on a default (an empty plan, a default winner) instead of its output.

    150/150 100%
  • Run completedharness.run-completed

    The environment was built, the architecture returned and grading finished, none of them throwing.

    150/150 100%
  • Structured replies arrivedharness.structured-output

    Every call that asked for structured output and ran to the end got it.

    150/150 100%

Harness health across every architecture →

A run of it

Plan-execute-verify on Return an expired, partially shipped order with Sonnet 5.5, repeat 1. It passed.

5 calls · 1m 20s · $0.177

  • tool call
  • denied by the guard
  • tool returned an error
  • shown no tools (reasoning only)
  • lost its tools (server failed)
  • failed

The whole run, message by message →

150 ranked runs of 150 recorded.

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.