ScaffoldBench

Harness health

Did every run measure what it claims?

A score says how a run went. It does not say whether the architecture it is filed under actually ran. Every run carries a list of checks that say so, and a run that fails one is excluded from every ranking and counted here instead.

Runs
900
in this run set, every model
Ranked
900
design verified, no API loss
Excluded: fidelity
0
the design did not run as designed
Excluded: API
0
lost to the API after retries
Retried
16
needed a second attempt

Which designs lost runs

An exclusion is the harness failing the design, not the design failing the task: a planner that returned no plan, a proposer that started without its tools. It is counted against the harness, and the run is left out so that it cannot flatter or sink the architecture it was filed under.

Every check, and how often it held

The runner’s own checks apply to every architecture. Each architecture adds checks that say what running as designed means for it.

The runner’s checks, on every run

  • No API failuresharness.api

    No model call failed at the API.

    900/900 100%

  • Builtin toolsharness.builtin-tools

    A check recorded with the run.

    900/900 100%

  • Every call endedharness.calls-ended

    Every call ended with a result, or was stopped by the run's own limits.

    900/900 100%

  • No harness tool deniedharness.harness-tools

    The sandbox guard never denied a harness tool, nor a tool the session itself offered.

    900/900 100%

  • Judge answeredharness.judge

    At least one grading call returned a usable verdict.

    108/108 100%

  • Tool servers connectedharness.mcp-connected

    Every tool server the harness gave a call was connected when the session started.

    900/900 100%

  • No foreign tool serversharness.mcp-hermetic

    No call loaded a tool server the harness did not give it.

    900/900 100%

  • No silent fallbacksharness.not-degraded

    No phase fell back on a default (an empty plan, a default winner) instead of its output.

    900/900 100%

  • Run completedharness.run-completed

    The environment was built, the architecture returned and grading finished, none of them throwing.

    900/900 100%

  • Structured replies arrivedharness.structured-output

    Every call that asked for structured output and ran to the end got it.

    900/900 100%

Single

  • Each call was shown the right toolssingle.tools

    The agent was shown exactly the task’s tools, none missing and none extra.

    150/150 100%

Plan-execute

  • The plan was a planplan-execute.plan

    The planner’s reply was read through its schema and held at least one step.

    150/150 100%

  • Each call was shown the right toolsplan-execute.tools

    The planner was shown no tools and the executor the task’s full toolset.

    150/150 100%

Orchestrator-workers

  • The task was decomposedorchestrator-workers.decomposition

    The orchestrator’s reply was read through its schema and held at least one subtask.

    150/150 100%

  • The fan-out happenedorchestrator-workers.fan-out

    Every subtask got a worker, every worker ran, and the recorded subagent count matches.

    150/150 100%

  • The synthesis wrote the answerorchestrator-workers.synthesis

    The final answer came from the synthesis call, not from a worker’s report standing in for it.

    150/150 100%

  • Each call was shown the right toolsorchestrator-workers.tools

    The orchestrator and the synthesis were shown no tools and every worker the task’s full toolset.

    150/150 100%

Plan-execute-verify

  • The plan was a planplan-execute-verify.plan

    The planner’s reply was read through its schema and held at least one step.

    150/150 100%

  • Every verdict was readplan-execute-verify.verdicts

    Every review round ended with a verdict read from the schema reply, not a default.

    150/150 100%

  • The loop did what its verdicts saidplan-execute-verify.loop

    Every verdict that called for a revision got one, and nothing was revised without such a verdict.

    150/150 100%

  • Each call was shown the right toolsplan-execute-verify.tools

    The planner was shown no tools, the verifier the read-only ones, and the executor and reviser the full toolset.

    150/150 100%

Propose-select

  • All three proposers ranpropose-select.proposals

    Three proposers ran in parallel, each from its own angle, and none was starved while another ran.

    150/150 100%

  • The selector chosepropose-select.selection

    The selector picked a proposal by a valid number, read from its schema reply and not a default.

    150/150 100%

  • Each call was shown the right toolspropose-select.tools

    Each proposer was shown the read-only tools, the selector none, and the executor the full toolset.

    150/150 100%

Gated-execute

  • Every state change was reviewedgated-execute.reviewed

    Every call that changed the world had been allowed by a review before it ran, counted from the world’s own journal.

    150/150 100%

  • Every decision was readgated-execute.decisions

    Every review that ran to its end gave its decision through the schema; an unreadable one is a denial.

    150/150 100%

  • Each call was shown the right toolsgated-execute.tools

    The executor was shown the task’s full toolset and each reviewer the read-only tools.

    150/150 100%

The runs that were left out

No run failed a check. Every recorded run was ranked, except as noted below.

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.