ScaffoldBench

Architecture

Plan-execute

Does committing to a sequence up front beat deciding one action at a time?

A tool-less planner writes the steps; a second call carries them out with the full toolset.

How it runs

  1. 1The planner

    no tools

    Sees the task and the domain policy but no tool, and returns an ordered list of steps through a structured reply. It cannot look anything up, so the plan is only as good as the statement of the task.

  2. 2The executor

    full toolset

    Receives the plan with the task and carries it out with the full toolset, adapting where the world differs from what the plan assumed.

Where it can fail quietly

An empty plan leaves a single agent with a wasted call in front of it, and the score cannot tell.

What the structure buys

Plan-execute against a single agent on the same tasks and repeats, on each model. The interval is on the paired difference, so it is as wide as the evidence is thin.

  • Haiku 4.5=1 of 6
    Score
    0.754
    0.634–0.862
    Cost / run
    $0.128
    1.5× single
    Ranked
    50/50

    not separable from single−0.017 [−0.143, +0.084]too few tasks carry the spread to show equivalence

  • Sonnet 5.5=1 of 6
    Score
    0.933
    0.895–0.966
    Cost / run
    $0.146
    1.3× single
    Ranked
    50/50

    not separable from single+0.034 [0.000, +0.087]too few tasks carry the spread to show equivalence

  • Opus 5.5=1 of 6
    Score
    0.947
    0.913–0.977
    Cost / run
    $0.336
    1.4× single
    Ranked
    50/50

    not separable from single+0.008 [−0.020, +0.039]too few tasks carry the spread to show equivalence

Task by task against a single agent

A mean can hide whether a design helps everywhere a little or on one task a lot. Each bar is the difference from a single agent on one task; the best-served tasks are at the top.

Haiku 4.5
Sonnet 5.5
Opus 5.5
Haiku 4.5: +0.33 against a single agent on Work a morning's customer inbox
Sonnet 5.5: level with a single agent on Work a morning's customer inbox
Opus 5.5: +0.24 against a single agent on Work a morning's customer inbox
Haiku 4.5: level with a single agent on Escalate a payroll outage the requester wants closed
Sonnet 5.5: +0.50 against a single agent on Escalate a payroll outage the requester wants closed
Opus 5.5: level with a single agent on Escalate a payroll outage the requester wants closed
Haiku 4.5: +0.23 against a single agent on Triage the weekend's queue on Monday morning
Sonnet 5.5: +0.07 against a single agent on Triage the weekend's queue on Monday morning
Opus 5.5: +0.07 against a single agent on Triage the weekend's queue on Monday morning
Haiku 4.5: +0.09 against a single agent on Run the major-incident runbook for a CI outage
Sonnet 5.5: +0.12 against a single agent on Run the major-incident runbook for a CI outage
Opus 5.5: +0.09 against a single agent on Run the major-incident runbook for a CI outage
Haiku 4.5: +0.29 against a single agent on Add checked bags and collect the right fee
Sonnet 5.5: level with a single agent on Add checked bags and collect the right fee
Opus 5.5: level with a single agent on Add checked bags and collect the right fee
Haiku 4.5: +0.29 against a single agent on Cancel a fare the customer wrongly believes is refundable to their card
Sonnet 5.5: level with a single agent on Cancel a fare the customer wrongly believes is refundable to their card
Opus 5.5: level with a single agent on Cancel a fare the customer wrongly believes is refundable to their card
Haiku 4.5: +0.12 against a single agent on Share two short deliveries out among backorders
Sonnet 5.5: +0.09 against a single agent on Share two short deliveries out among backorders
Opus 5.5: level with a single agent on Share two short deliveries out among backorders
Haiku 4.5: +0.11 against a single agent on Audit a week of baggage fees
Sonnet 5.5: +0.01 against a single agent on Audit a week of baggage fees
Opus 5.5: +0.07 against a single agent on Audit a week of baggage fees
Haiku 4.5: +0.10 against a single agent on OWASP vs NIST on password hashing
Sonnet 5.5: +0.03 against a single agent on OWASP vs NIST on password hashing
Opus 5.5: level with a single agent on OWASP vs NIST on password hashing
Haiku 4.5: −0.01 against a single agent on Work a cancellation queue where cash cannot be recalled
Sonnet 5.5: +0.10 against a single agent on Work a cancellation queue where cash cannot be recalled
Opus 5.5: level with a single agent on Work a cancellation queue where cash cannot be recalled
Haiku 4.5: −0.09 against a single agent on Work the access request queue
Sonnet 5.5: +0.04 against a single agent on Work the access request queue
Opus 5.5: +0.09 against a single agent on Work the access request queue
Haiku 4.5: +0.04 against a single agent on Settle a desk of returns by what actually came back
Sonnet 5.5: level with a single agent on Settle a desk of returns by what actually came back
Opus 5.5: level with a single agent on Settle a desk of returns by what actually came back
Haiku 4.5: +0.01 against a single agent on Re-protect a cancelled flight when seats are scarce
Sonnet 5.5: level with a single agent on Re-protect a cancelled flight when seats are scarce
Opus 5.5: level with a single agent on Re-protect a cancelled flight when seats are scarce
Haiku 4.5: level with a single agent on Redirect the order that has not shipped
Sonnet 5.5: level with a single agent on Redirect the order that has not shipped
Opus 5.5: level with a single agent on Redirect the order that has not shipped
Haiku 4.5: level with a single agent on Return an expired, partially shipped order
Sonnet 5.5: level with a single agent on Return an expired, partially shipped order
Opus 5.5: level with a single agent on Return an expired, partially shipped order
Haiku 4.5: level with a single agent on Cancel one line of a multi-line order
Sonnet 5.5: level with a single agent on Cancel one line of a multi-line order
Opus 5.5: level with a single agent on Cancel one line of a multi-line order
Haiku 4.5: level with a single agent on HTTP 301 vs 308
Sonnet 5.5: level with a single agent on HTTP 301 vs 308
Opus 5.5: level with a single agent on HTTP 301 vs 308
Haiku 4.5: level with a single agent on Triage a new service desk ticket
Sonnet 5.5: level with a single agent on Triage a new service desk ticket
Opus 5.5: level with a single agent on Triage a new service desk ticket
Haiku 4.5: +0.03 against a single agent on Settle change requests under dated bulletins
Sonnet 5.5: −0.05 against a single agent on Settle change requests under dated bulletins
Opus 5.5: level with a single agent on Settle change requests under dated bulletins
Haiku 4.5: level with a single agent on async vs defer on <script>
Sonnet 5.5: level with a single agent on async vs defer on <script>
Opus 5.5: −0.04 against a single agent on async vs defer on <script>
Haiku 4.5: level with a single agent on Twelve HTTP status codes, by the book
Sonnet 5.5: level with a single agent on Twelve HTTP status codes, by the book
Opus 5.5: −0.08 against a single agent on Twelve HTTP status codes, by the book
Haiku 4.5: level with a single agent on NIST SP 800-63B password rules, current revision
Sonnet 5.5: level with a single agent on NIST SP 800-63B password rules, current revision
Opus 5.5: −0.13 against a single agent on NIST SP 800-63B password rules, current revision
Haiku 4.5: −0.38 against a single agent on Where ten HTTP header fields are defined
Sonnet 5.5: level with a single agent on Where ten HTTP header fields are defined
Opus 5.5: −0.05 against a single agent on Where ten HTTP header fields are defined
Haiku 4.5: −0.58 against a single agent on Consolidate duplicate CRM reports
Sonnet 5.5: −0.08 against a single agent on Consolidate duplicate CRM reports
Opus 5.5: −0.08 against a single agent on Consolidate duplicate CRM reports
Haiku 4.5: −1.00 against a single agent on Rebook onto the earliest available flight and settle the difference
Sonnet 5.5: level with a single agent on Rebook onto the earliest available flight and settle the difference
Opus 5.5: level with a single agent on Rebook onto the earliest available flight and settle the difference
Mean score of the architecture minus a single agent’s, over the repeats both completed, on a shared zero from −1 to +1. Runs that were excluded are left out.

Where the money goes

A run of Plan-execute is 2 stages. This is what each one costs, so the price of the structure is separate from the price of the work.

  • Haiku 4.5$0.128 per run · 50 runs
  • Sonnet 5.5$0.146 per run · 50 runs
  • Opus 5.5$0.336 per run · 50 runs
StageHaiku 4.5Sonnet 5.5Opus 5.5
The planner$0.02519%$0.01712%$0.04112%
The executor$0.10481%$0.13088%$0.29488%
Mean cost of a ranked run, by the stage of the design that spent it. Judge calls are not counted.

Did it run as designed?

Each check is recorded with every run. A run that fails one is excluded from every ranking and counted, never scored.

The design’s own checks

  • The plan was a planplan-execute.plan

    The planner’s reply was read through its schema and held at least one step.

    150/150 100%
  • Each call was shown the right toolsplan-execute.tools

    The planner was shown no tools and the executor the task’s full toolset.

    150/150 100%
The harness’s checksall 10 held in every run
  • No API failuresharness.api

    No model call failed at the API.

    150/150 100%
  • Builtin toolsharness.builtin-tools

    A check recorded with the run.

    150/150 100%
  • Every call endedharness.calls-ended

    Every call ended with a result, or was stopped by the run's own limits.

    150/150 100%
  • No harness tool deniedharness.harness-tools

    The sandbox guard never denied a harness tool, nor a tool the session itself offered.

    150/150 100%
  • Judge answeredharness.judge

    At least one grading call returned a usable verdict.

    18/18 100%
  • Tool servers connectedharness.mcp-connected

    Every tool server the harness gave a call was connected when the session started.

    150/150 100%
  • No foreign tool serversharness.mcp-hermetic

    No call loaded a tool server the harness did not give it.

    150/150 100%
  • No silent fallbacksharness.not-degraded

    No phase fell back on a default (an empty plan, a default winner) instead of its output.

    150/150 100%
  • Run completedharness.run-completed

    The environment was built, the architecture returned and grading finished, none of them throwing.

    150/150 100%
  • Structured replies arrivedharness.structured-output

    Every call that asked for structured output and ran to the end got it.

    150/150 100%

Harness health across every architecture →

A run of it

Plan-execute on Cancel a fare the customer wrongly believes is refundable to their card with Haiku 4.5, repeat 1. It passed.

2 calls · 1m 32s · $0.069

  • tool call
  • denied by the guard
  • tool returned an error
  • shown no tools (reasoning only)
  • lost its tools (server failed)
  • failed

The whole run, message by message →

150 ranked runs of 150 recorded.

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.