Architecture
Plan-execute
Does committing to a sequence up front beat deciding one action at a time?
A tool-less planner writes the steps; a second call carries them out with the full toolset.
How it runs
1The planner
no tools
Sees the task and the domain policy but no tool, and returns an ordered list of steps through a structured reply. It cannot look anything up, so the plan is only as good as the statement of the task.
2The executor
full toolset
Receives the plan with the task and carries it out with the full toolset, adapting where the world differs from what the plan assumed.
Where it can fail quietly
An empty plan leaves a single agent with a wasted call in front of it, and the score cannot tell.
What the structure buys
Plan-execute against a single agent on the same tasks and repeats, on each model. The interval is on the paired difference, so it is as wide as the evidence is thin.
- Haiku 4.5=1 of 6
- Score
- 0.754
- 0.634–0.862
- Cost / run
- $0.128
- 1.5× single
- Ranked
- 50/50
not separable from single−0.017 [−0.143, +0.084]too few tasks carry the spread to show equivalence
- Sonnet 5.5=1 of 6
- Score
- 0.933
- 0.895–0.966
- Cost / run
- $0.146
- 1.3× single
- Ranked
- 50/50
not separable from single+0.034 [0.000, +0.087]too few tasks carry the spread to show equivalence
- Opus 5.5=1 of 6
- Score
- 0.947
- 0.913–0.977
- Cost / run
- $0.336
- 1.4× single
- Ranked
- 50/50
not separable from single+0.008 [−0.020, +0.039]too few tasks carry the spread to show equivalence
Task by task against a single agent
A mean can hide whether a design helps everywhere a little or on one task a lot. Each bar is the difference from a single agent on one task; the best-served tasks are at the top.
Where the money goes
A run of Plan-execute is 2 stages. This is what each one costs, so the price of the structure is separate from the price of the work.
- Haiku 4.5$0.128 per run · 50 runs
- Sonnet 5.5$0.146 per run · 50 runs
- Opus 5.5$0.336 per run · 50 runs
| Stage | Haiku 4.5 | Sonnet 5.5 | Opus 5.5 |
|---|---|---|---|
| The planner | $0.02519% | $0.01712% | $0.04112% |
| The executor | $0.10481% | $0.13088% | $0.29488% |
Did it run as designed?
Each check is recorded with every run. A run that fails one is excluded from every ranking and counted, never scored.
The design’s own checks
The plan was a plan
plan-execute.planThe planner’s reply was read through its schema and held at least one step.
150/150 100%Each call was shown the right tools
plan-execute.toolsThe planner was shown no tools and the executor the task’s full toolset.
150/150 100%
The harness’s checksall 10 held in every run
No API failures
harness.apiNo model call failed at the API.
150/150 100%Builtin tools
harness.builtin-toolsA check recorded with the run.
150/150 100%Every call ended
harness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
150/150 100%No harness tool denied
harness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
150/150 100%Judge answered
harness.judgeAt least one grading call returned a usable verdict.
18/18 100%Tool servers connected
harness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
150/150 100%No foreign tool servers
harness.mcp-hermeticNo call loaded a tool server the harness did not give it.
150/150 100%No silent fallbacks
harness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
150/150 100%Run completed
harness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
150/150 100%Structured replies arrived
harness.structured-outputEvery call that asked for structured output and ran to the end got it.
150/150 100%
A run of it
Plan-execute on Cancel a fare the customer wrongly believes is refundable to their card with Haiku 4.5, repeat 1. It passed.
2 calls · 1m 32s · $0.069
- tool call
- denied by the guard
- tool returned an error
- shown no tools (reasoning only)
- lost its tools (server failed)
- failed
150 ranked runs of 150 recorded.
Provenance of the numbers on this page
- run set
- scaffold-or-scale-v1
- models
- Haiku 4.5, Sonnet 5.5, Opus 5.5
- effort
- high
- repeats
- 2
- fixtures
- replay
- runs
- 900 / 900
- strategy cost
- $354.44
- judge cost
- $9.19
- wall clock
- 9h 48m
- git
- fce5807b2d
Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.