ScaffoldBench

Architecture

Gated-execute

Is review at the point of action worth a model call per action?

One agent acts, but every state-changing tool call must pass a read-only reviewer before it runs.

How it runs

  1. 1The executor

    full toolset

    One agent with the full toolset. Reads pass straight through; every call that would change state is held until a reviewer has ruled on it.

  2. 2The reviewer

    read-only tools

    A separate call per state-changing action. It holds the domain policy and the read-only tools, checks the current state and allows or denies. A denial returns to the executor as the tool’s result, with the reason, so it can adapt.

Where it can fail quietly

If a state-changing call can run without a review the gate is decoration, and an unreadable decision must count as a denial.

What the structure buys

Gated-execute against a single agent on the same tasks and repeats, on each model. The interval is on the paired difference, so it is as wide as the evidence is thin.

  • Haiku 4.5=1 of 6
    Score
    0.677
    0.534–0.809
    Cost / run
    $0.862
    4.8× single
    Ranked
    50/50

    not separable from single−0.094 [−0.215, +0.005]too few tasks carry the spread to show equivalence

  • Sonnet 5.5=1 of 6
    Score
    0.927
    0.889–0.963
    Cost / run
    $0.495
    2.6× single
    Ranked
    50/50

    not separable from single+0.028 [−0.012, +0.083]too few tasks carry the spread to show equivalence

  • Opus 5.5=1 of 6
    Score
    0.956
    0.922–0.985
    Cost / run
    $0.934
    2.5× single
    Ranked
    50/50

    not separable from single+0.016 [0.000, +0.041]too few tasks carry the spread to show equivalence

Task by task against a single agent

A mean can hide whether a design helps everywhere a little or on one task a lot. Each bar is the difference from a single agent on one task; the best-served tasks are at the top.

Haiku 4.5
Sonnet 5.5
Opus 5.5
Haiku 4.5: level with a single agent on Escalate a payroll outage the requester wants closed
Sonnet 5.5: +0.50 against a single agent on Escalate a payroll outage the requester wants closed
Opus 5.5: level with a single agent on Escalate a payroll outage the requester wants closed
Haiku 4.5: +0.08 against a single agent on Share two short deliveries out among backorders
Sonnet 5.5: +0.22 against a single agent on Share two short deliveries out among backorders
Opus 5.5: level with a single agent on Share two short deliveries out among backorders
Haiku 4.5: +0.29 against a single agent on Add checked bags and collect the right fee
Sonnet 5.5: level with a single agent on Add checked bags and collect the right fee
Opus 5.5: level with a single agent on Add checked bags and collect the right fee
Haiku 4.5: +0.17 against a single agent on Triage the weekend's queue on Monday morning
Sonnet 5.5: +0.02 against a single agent on Triage the weekend's queue on Monday morning
Opus 5.5: +0.05 against a single agent on Triage the weekend's queue on Monday morning
Haiku 4.5: level with a single agent on NIST SP 800-63B password rules, current revision
Sonnet 5.5: level with a single agent on NIST SP 800-63B password rules, current revision
Opus 5.5: +0.13 against a single agent on NIST SP 800-63B password rules, current revision
Haiku 4.5: +0.11 against a single agent on Work a cancellation queue where cash cannot be recalled
Sonnet 5.5: level with a single agent on Work a cancellation queue where cash cannot be recalled
Opus 5.5: level with a single agent on Work a cancellation queue where cash cannot be recalled
Haiku 4.5: +0.03 against a single agent on HTTP 301 vs 308
Sonnet 5.5: +0.06 against a single agent on HTTP 301 vs 308
Opus 5.5: level with a single agent on HTTP 301 vs 308
Haiku 4.5: +0.08 against a single agent on Run the major-incident runbook for a CI outage
Sonnet 5.5: level with a single agent on Run the major-incident runbook for a CI outage
Opus 5.5: level with a single agent on Run the major-incident runbook for a CI outage
Haiku 4.5: +0.08 against a single agent on Settle a desk of returns by what actually came back
Sonnet 5.5: level with a single agent on Settle a desk of returns by what actually came back
Opus 5.5: level with a single agent on Settle a desk of returns by what actually came back
Haiku 4.5: +0.03 against a single agent on Work a morning's customer inbox
Sonnet 5.5: level with a single agent on Work a morning's customer inbox
Opus 5.5: level with a single agent on Work a morning's customer inbox
Haiku 4.5: level with a single agent on Rebook onto the earliest available flight and settle the difference
Sonnet 5.5: level with a single agent on Rebook onto the earliest available flight and settle the difference
Opus 5.5: level with a single agent on Rebook onto the earliest available flight and settle the difference
Haiku 4.5: level with a single agent on Redirect the order that has not shipped
Sonnet 5.5: level with a single agent on Redirect the order that has not shipped
Opus 5.5: level with a single agent on Redirect the order that has not shipped
Haiku 4.5: level with a single agent on Cancel one line of a multi-line order
Sonnet 5.5: level with a single agent on Cancel one line of a multi-line order
Opus 5.5: level with a single agent on Cancel one line of a multi-line order
Haiku 4.5: level with a single agent on Where ten HTTP header fields are defined
Sonnet 5.5: level with a single agent on Where ten HTTP header fields are defined
Opus 5.5: level with a single agent on Where ten HTTP header fields are defined
Haiku 4.5: level with a single agent on Twelve HTTP status codes, by the book
Sonnet 5.5: level with a single agent on Twelve HTTP status codes, by the book
Opus 5.5: level with a single agent on Twelve HTTP status codes, by the book
Haiku 4.5: level with a single agent on Triage a new service desk ticket
Sonnet 5.5: level with a single agent on Triage a new service desk ticket
Opus 5.5: level with a single agent on Triage a new service desk ticket
Haiku 4.5: −0.02 against a single agent on OWASP vs NIST on password hashing
Sonnet 5.5: level with a single agent on OWASP vs NIST on password hashing
Opus 5.5: level with a single agent on OWASP vs NIST on password hashing
Haiku 4.5: −0.05 against a single agent on async vs defer on <script>
Sonnet 5.5: −0.05 against a single agent on async vs defer on <script>
Opus 5.5: level with a single agent on async vs defer on <script>
Haiku 4.5: −0.44 against a single agent on Audit a week of baggage fees
Sonnet 5.5: +0.05 against a single agent on Audit a week of baggage fees
Opus 5.5: +0.23 against a single agent on Audit a week of baggage fees
Haiku 4.5: −0.21 against a single agent on Cancel a fare the customer wrongly believes is refundable to their card
Sonnet 5.5: level with a single agent on Cancel a fare the customer wrongly believes is refundable to their card
Opus 5.5: level with a single agent on Cancel a fare the customer wrongly believes is refundable to their card
Haiku 4.5: −0.35 against a single agent on Consolidate duplicate CRM reports
Sonnet 5.5: +0.08 against a single agent on Consolidate duplicate CRM reports
Opus 5.5: level with a single agent on Consolidate duplicate CRM reports
Haiku 4.5: −0.36 against a single agent on Settle change requests under dated bulletins
Sonnet 5.5: level with a single agent on Settle change requests under dated bulletins
Opus 5.5: level with a single agent on Settle change requests under dated bulletins
Haiku 4.5: −0.37 against a single agent on Work the access request queue
Sonnet 5.5: level with a single agent on Work the access request queue
Opus 5.5: level with a single agent on Work the access request queue
Haiku 4.5: −0.44 against a single agent on Re-protect a cancelled flight when seats are scarce
Sonnet 5.5: level with a single agent on Re-protect a cancelled flight when seats are scarce
Opus 5.5: level with a single agent on Re-protect a cancelled flight when seats are scarce
Haiku 4.5: −1.00 against a single agent on Return an expired, partially shipped order
Sonnet 5.5: −0.18 against a single agent on Return an expired, partially shipped order
Opus 5.5: level with a single agent on Return an expired, partially shipped order
Mean score of the architecture minus a single agent’s, over the repeats both completed, on a shared zero from −1 to +1. Runs that were excluded are left out.

Where the money goes

A run of Gated-execute is 2 stages. This is what each one costs, so the price of the structure is separate from the price of the work.

  • Haiku 4.5$0.862 per run · 50 runs
  • Sonnet 5.5$0.495 per run · 50 runs
  • Opus 5.5$0.934 per run · 50 runs
StageHaiku 4.5Sonnet 5.5Opus 5.5
The executor$0.14216%$0.11724%$0.24626%
The reviewer$0.72084%$0.37876%$0.68874%
Mean cost of a ranked run, by the stage of the design that spent it. Judge calls are not counted.

Did it run as designed?

Each check is recorded with every run. A run that fails one is excluded from every ranking and counted, never scored.

The design’s own checks

  • Every state change was reviewedgated-execute.reviewed

    Every call that changed the world had been allowed by a review before it ran, counted from the world’s own journal.

    150/150 100%
  • Every decision was readgated-execute.decisions

    Every review that ran to its end gave its decision through the schema; an unreadable one is a denial.

    150/150 100%
  • Each call was shown the right toolsgated-execute.tools

    The executor was shown the task’s full toolset and each reviewer the read-only tools.

    150/150 100%
The harness’s checksall 10 held in every run
  • No API failuresharness.api

    No model call failed at the API.

    150/150 100%
  • Builtin toolsharness.builtin-tools

    A check recorded with the run.

    150/150 100%
  • Every call endedharness.calls-ended

    Every call ended with a result, or was stopped by the run's own limits.

    150/150 100%
  • No harness tool deniedharness.harness-tools

    The sandbox guard never denied a harness tool, nor a tool the session itself offered.

    150/150 100%
  • Judge answeredharness.judge

    At least one grading call returned a usable verdict.

    18/18 100%
  • Tool servers connectedharness.mcp-connected

    Every tool server the harness gave a call was connected when the session started.

    150/150 100%
  • No foreign tool serversharness.mcp-hermetic

    No call loaded a tool server the harness did not give it.

    150/150 100%
  • No silent fallbacksharness.not-degraded

    No phase fell back on a default (an empty plan, a default winner) instead of its output.

    150/150 100%
  • Run completedharness.run-completed

    The environment was built, the architecture returned and grading finished, none of them throwing.

    150/150 100%
  • Structured replies arrivedharness.structured-output

    Every call that asked for structured output and ran to the end got it.

    150/150 100%

Harness health across every architecture →

A run of it

Gated-execute on Cancel a fare the customer wrongly believes is refundable to their card with Haiku 4.5, repeat 1. It scored 0.000.

3 calls · 1m 17s · $0.086

  • tool call
  • denied by the guard
  • tool returned an error
  • shown no tools (reasoning only)
  • lost its tools (server failed)
  • failed

The whole run, message by message →

150 ranked runs of 150 recorded.

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.