Architecture
Single
Is any structure worth its price at all?
One agent, one context, the full toolset. The control condition every other architecture has to beat.
How it runs
1The agent
full toolset
Reads the task, calls tools as it sees fit and answers in one session. Every other design adds calls around this one, so its score and cost are what each of them has to beat.
Where it can fail quietly
Nothing to skip: the control cannot degrade in a way that makes it look like something else.
What the structure buys
The control. Every other architecture is measured as a difference from these numbers, task by task and repeat by repeat.
- Haiku 4.5=1 of 6
- Score
- 0.771
- 0.673–0.868
- Cost / run
- $0.091
- baseline
- Ranked
- 50/50
the control
- Sonnet 5.5=1 of 6
- Score
- 0.900
- 0.843–0.947
- Cost / run
- $0.116
- baseline
- Ranked
- 50/50
the control
- Opus 5.5=1 of 6
- Score
- 0.939
- 0.903–0.974
- Cost / run
- $0.252
- baseline
- Ranked
- 50/50
the control
Where the money goes
A run of Single is one call. This is what each one costs, so the price of the structure is separate from the price of the work.
- Haiku 4.5$0.091 per run · 50 runs
- Sonnet 5.5$0.116 per run · 50 runs
- Opus 5.5$0.252 per run · 50 runs
| Stage | Haiku 4.5 | Sonnet 5.5 | Opus 5.5 |
|---|---|---|---|
| The agent | $0.091100% | $0.116100% | $0.252100% |
Did it run as designed?
Each check is recorded with every run. A run that fails one is excluded from every ranking and counted, never scored.
The design’s own checks
Each call was shown the right tools
single.toolsThe agent was shown exactly the task’s tools, none missing and none extra.
150/150 100%
The harness’s checksall 10 held in every run
No API failures
harness.apiNo model call failed at the API.
150/150 100%Builtin tools
harness.builtin-toolsA check recorded with the run.
150/150 100%Every call ended
harness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
150/150 100%No harness tool denied
harness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
150/150 100%Judge answered
harness.judgeAt least one grading call returned a usable verdict.
18/18 100%Tool servers connected
harness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
150/150 100%No foreign tool servers
harness.mcp-hermeticNo call loaded a tool server the harness did not give it.
150/150 100%No silent fallbacks
harness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
150/150 100%Run completed
harness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
150/150 100%Structured replies arrived
harness.structured-outputEvery call that asked for structured output and ran to the end got it.
150/150 100%
A run of it
Single on Cancel a fare the customer wrongly believes is refundable to their card with Haiku 4.5, repeat 1. It passed.
1 call · 26.0s · $0.031
- tool call
- denied by the guard
- tool returned an error
- shown no tools (reasoning only)
- lost its tools (server failed)
- failed
150 ranked runs of 150 recorded.
Provenance of the numbers on this page
- run set
- scaffold-or-scale-v1
- models
- Haiku 4.5, Sonnet 5.5, Opus 5.5
- effort
- high
- repeats
- 2
- fixtures
- replay
- runs
- 900 / 900
- strategy cost
- $354.44
- judge cost
- $9.19
- wall clock
- 9h 48m
- git
- fce5807b2d
Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.