ScaffoldBench

Architecture

Single

Is any structure worth its price at all?

One agent, one context, the full toolset. The control condition every other architecture has to beat.

How it runs

  1. 1The agent

    full toolset

    Reads the task, calls tools as it sees fit and answers in one session. Every other design adds calls around this one, so its score and cost are what each of them has to beat.

Where it can fail quietly

Nothing to skip: the control cannot degrade in a way that makes it look like something else.

What the structure buys

The control. Every other architecture is measured as a difference from these numbers, task by task and repeat by repeat.

  • Haiku 4.5=1 of 6
    Score
    0.771
    0.673–0.868
    Cost / run
    $0.091
    baseline
    Ranked
    50/50

    the control

  • Sonnet 5.5=1 of 6
    Score
    0.900
    0.843–0.947
    Cost / run
    $0.116
    baseline
    Ranked
    50/50

    the control

  • Opus 5.5=1 of 6
    Score
    0.939
    0.903–0.974
    Cost / run
    $0.252
    baseline
    Ranked
    50/50

    the control

Where the money goes

A run of Single is one call. This is what each one costs, so the price of the structure is separate from the price of the work.

  • Haiku 4.5$0.091 per run · 50 runs
  • Sonnet 5.5$0.116 per run · 50 runs
  • Opus 5.5$0.252 per run · 50 runs
StageHaiku 4.5Sonnet 5.5Opus 5.5
The agent$0.091100%$0.116100%$0.252100%
Mean cost of a ranked run, by the stage of the design that spent it. Judge calls are not counted.

Did it run as designed?

Each check is recorded with every run. A run that fails one is excluded from every ranking and counted, never scored.

The design’s own checks

  • Each call was shown the right toolssingle.tools

    The agent was shown exactly the task’s tools, none missing and none extra.

    150/150 100%
The harness’s checksall 10 held in every run
  • No API failuresharness.api

    No model call failed at the API.

    150/150 100%
  • Builtin toolsharness.builtin-tools

    A check recorded with the run.

    150/150 100%
  • Every call endedharness.calls-ended

    Every call ended with a result, or was stopped by the run's own limits.

    150/150 100%
  • No harness tool deniedharness.harness-tools

    The sandbox guard never denied a harness tool, nor a tool the session itself offered.

    150/150 100%
  • Judge answeredharness.judge

    At least one grading call returned a usable verdict.

    18/18 100%
  • Tool servers connectedharness.mcp-connected

    Every tool server the harness gave a call was connected when the session started.

    150/150 100%
  • No foreign tool serversharness.mcp-hermetic

    No call loaded a tool server the harness did not give it.

    150/150 100%
  • No silent fallbacksharness.not-degraded

    No phase fell back on a default (an empty plan, a default winner) instead of its output.

    150/150 100%
  • Run completedharness.run-completed

    The environment was built, the architecture returned and grading finished, none of them throwing.

    150/150 100%
  • Structured replies arrivedharness.structured-output

    Every call that asked for structured output and ran to the end got it.

    150/150 100%

Harness health across every architecture →

A run of it

Single on Cancel a fare the customer wrongly believes is refundable to their card with Haiku 4.5, repeat 1. It passed.

1 call · 26.0s · $0.031

  • tool call
  • denied by the guard
  • tool returned an error
  • shown no tools (reasoning only)
  • lost its tools (server failed)
  • failed

The whole run, message by message →

150 ranked runs of 150 recorded.

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.