How the numbers are made.

ScaffoldBench is a grid of model, architecture, task and repeat. The run set published here is 3 models (Haiku 4.5, Sonnet 5.5, Opus 5.5) × 6 architectures × 25 tasks × 2 repeats: 900 runs. In every cell the only thing that changes is the harness around the model. The model, the tools the task offers, the task prompt, the domain policy appended to the system prompt, the reasoning effort and the cap on cost and wall-clock time per run are the same for every architecture.

The six architectures are on the designs page, each drawn as the calls it makes. The first of them, single, is the control: one agent with the full toolset. Every other architecture is measured as a difference from it, because the question is never whether an architecture works but whether its extra calls earn their cost.

Tool-use tasks are graded by code. A task lists weighted assertions about the final state of its simulated world and about the journal of tool calls the agent made, such as whether it read the refund policy before it moved money. Rules add penalties that multiply the score, so a broken hard rule zeroes a run however well the rest went. A run passes at a score of 0.999 or more. No model decides who won.

Research tasks are graded by a judge model against a rubric. Each item is scored several times and the median taken, so one unlucky grading cannot move a score. The score adds the coverage of the sources a thorough answer should have consulted, and an audit of the citations: an answer that cites a page the agent never fetched is penalised. The judge’s cost is tracked apart from the run’s, so a scoring model never flatters or penalises an architecture. In replay mode, research tasks read a recorded corpus of pages rather than the live web, so a run can be repeated.

A task may also carry a difficulty, a hypothesis written before it was run (which architecture should do best, and why) and its traps: the specific ways a run can go wrong, each tied to the assertions that register it. The matrix and the task pages show them.

The first run set tied five architectures because four of them never ran as designed, and nothing checked. The post-mortem tells it. Now every run carries a list of checks. The runner’s own say the harness did its job: no API failure, no harness tool denied, every tool server connected and no foreign one, no silent fallback on a default, every structured reply received. Each architecture adds checks that say what running as designed means for it: a plan with at least one step, a fan-out that happened, a verdict that was read, a selector that chose.

A run that fails a check is excluded from every ranking and counted, never scored. A run lost to the API after the runner’s retries is excluded too, because an outage is not an architecture’s failure. Exclusions are not hidden: they are counted at the top of every results page, and every excluded run and the check that failed is listed on the harness health page. A run set recorded without checks is shown but ranked nowhere.

The runner’s checks are in packages/core/src/integrity.ts; each architecture’s are beside its strategy in packages/strategies/src.

A benchmark this size overstates differences far more often than it misses them, so every choice below is there to keep a ranking honest. When the data cannot separate two architectures, the site says so.

• The task is the unit. The repeats of one task resemble each other much more than they resemble another task, so they are not independent evidence. Every interval resamples whole tasks, keeping each task’s repeats together.
• Comparisons are paired. Two architectures are compared on the same task and the same repeat, which removes the large variance between tasks. A pass rate carries a Wilson score interval, computed on the number of runs that are effectively independent once tasks are accounted for; cost and latency are compared as ratios, so “better” means cheaper or faster.
• Four verdicts, not two. A comparison ends better, worse, equivalent within a margin, or not separable. “Not separable” is a finding about the data, not a hidden tie. With fewer than 5 tasks no verdict is drawn, and where every task moved by the same amount there is no interval to show: the table says “no interval” and why.
• Many comparisons are paid for. Each family of verdicts, one kind of comparison on one metric across every model, is adjusted for the comparisons it makes at once (Holm), so the chance of any false call in the family stays at 5%. The intervals shown are the plain 95% ones, so an interval can exclude zero while the adjusted verdict stays “not separable”.
• The margin is fixed in advance. “Equivalent” means the difference is shown, by two one-sided tests, to lie within ±0.05 of score (×/÷1.1 of cost), a margin chosen before the data was seen.

The derivation of each step, and how the intervals behave in simulation, is in docs/STATISTICS.md.

Ties are shown as ties. Two architectures share a rank when neither is reliably above anything the other is not. The leaderboard writes =1 and groups them, rather than ordering them by a difference the data does not support.

Scaffold or scale? To ask whether structure beats a bigger model, an architecture on one model is compared, task by task, with a plain single agent on the next model up. A scaffold that scores reliably higher at a lower cost has bought more than the upgrade would, for less. The comparison is drawn for every architecture, not only the best one, because picking the winner from the same data and then testing it flatters it.

Before the flagship run starts, its hypotheses, metrics, margins and decision rules are committed to docs/PREREGISTRATION.md. The run set’s manifest records the commit it ran at, so anyone can check that the commitment came first. Each task’s own hypothesis, the architecture it expects to do best and why, is written when the task is, and is shown on its page beside what happened.

A hypothesis that the data contradicts is reported as contradicted. That is the point of writing it down.

• Tasks like these, not agents in general. The tasks are hand-written, in a few simulated worlds and a research set, by one author. Their difficulty labels are the author’s, checked by pilot runs. The intervals say how sure the data is about these kinds of task.
• One model family, one SDK. Every model is a Claude model run through the Claude Agent SDK. Another family, or another harness runtime, may rank the designs differently.
• Prompts are a confound. We wrote each architecture’s prompts. Every design gets the same task prompt and domain policy, which narrows the gap, but a better prompt for one design could move its standing.
• Structure and tokens are not separated. A design that makes more calls spends more tokens, and there is no compute-matched control yet. Read a gain beside its cost ratio, never alone.
• Cost is what the SDK reported, not an invoice. Prompt caching varies from run to run, and latency is wall-clock time with other runs in flight.
• Few repeats. With a handful of repeats per task, small differences stay not separable. That is the site refusing to guess, not an absence of effect.
• The judge can be wrong. Research scores rest on a model’s grading. Taking the median of several samples reduces its noise; it does not remove its bias.
• A fidelity check proves structure, not quality. It shows that a plan existed and a verdict was read. It does not show that the plan was good.
• Models change. A result is dated to the run set’s models and commit, both recorded in its manifest.

Everything on this site is read from results/ at build time, and the records are plain JSON. To run a cell yourself, from the repository root (it spends tokens on whatever Claude account your shell is signed in to; --update-latest is what makes the site show the new run set):

npm ci
npm run bench -- run --strategies single,plan-execute --tasks airline-hard --repeats 1 --update-latest
npm run dev:web

Adding a task is described in docs/ADDING-A-TASK.md, and the shape of the code in docs/ARCHITECTURE.md.