Architecture
Plan-execute-verify
Is a second opinion on the finished state worth the cost of asking for it?
Plan and execute, then a verifier with read-only tools checks the real state and can send the work back.
How it runs
1The planner
no tools
The same tool-less planner as in plan-execute: an ordered list of steps through a structured reply.
2The executor
full toolset
Carries out the plan with the full toolset. When the verifier sends the work back, a revision call, with the full toolset again, acts on its issues, at most twice.
3The verifier
read-only tools
Holds only the read-only tools and has to check the real state of the world rather than the executor’s account of it. It returns a verdict with a severity: a blocking issue sends the work back, a minor one ends the review.
Where it can fail quietly
A verifier whose verdict cannot be read, or that approves by default, never sends anything back: the loop is dead code and the cost is overhead.
What the structure buys
Plan-execute-verify against a single agent on the same tasks and repeats, on each model. The interval is on the paired difference, so it is as wide as the evidence is thin.
- Haiku 4.5=1 of 6
- Score
- 0.723
- 0.588–0.845
- Cost / run
- $0.305
- 3.3× single
- Ranked
- 50/50
not separable from single−0.048 [−0.171, +0.051]too few tasks carry the spread to show equivalence
- Sonnet 5.5=1 of 6
- Score
- 0.930
- 0.856–0.985
- Cost / run
- $0.278
- 2.4× single
- Ranked
- 50/50
not separable from single+0.031 [−0.019, +0.077]too few tasks carry the spread to show equivalence
- Opus 5.5=1 of 6
- Score
- 0.947
- 0.890–0.986
- Cost / run
- $0.561
- 2.4× single
- Ranked
- 50/50
not separable from single+0.008 [−0.042, +0.053]too few tasks carry the spread to show equivalence
Task by task against a single agent
A mean can hide whether a design helps everywhere a little or on one task a lot. Each bar is the difference from a single agent on one task; the best-served tasks are at the top.
Where the money goes
A run of Plan-execute-verify is 3 stages. This is what each one costs, so the price of the structure is separate from the price of the work.
- Haiku 4.5$0.305 per run · 50 runs
- Sonnet 5.5$0.278 per run · 50 runs
- Opus 5.5$0.561 per run · 50 runs
| Stage | Haiku 4.5 | Sonnet 5.5 | Opus 5.5 |
|---|---|---|---|
| The planner | $0.0258% | $0.0155% | $0.0346% |
| The executor | $0.14146% | $0.15656% | $0.31757% |
| The verifier | $0.14046% | $0.10638% | $0.20937% |
Did it run as designed?
Each check is recorded with every run. A run that fails one is excluded from every ranking and counted, never scored.
The design’s own checks
The plan was a plan
plan-execute-verify.planThe planner’s reply was read through its schema and held at least one step.
150/150 100%Every verdict was read
plan-execute-verify.verdictsEvery review round ended with a verdict read from the schema reply, not a default.
150/150 100%The loop did what its verdicts said
plan-execute-verify.loopEvery verdict that called for a revision got one, and nothing was revised without such a verdict.
150/150 100%Each call was shown the right tools
plan-execute-verify.toolsThe planner was shown no tools, the verifier the read-only ones, and the executor and reviser the full toolset.
150/150 100%
The harness’s checksall 10 held in every run
No API failures
harness.apiNo model call failed at the API.
150/150 100%Builtin tools
harness.builtin-toolsA check recorded with the run.
150/150 100%Every call ended
harness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
150/150 100%No harness tool denied
harness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
150/150 100%Judge answered
harness.judgeAt least one grading call returned a usable verdict.
18/18 100%Tool servers connected
harness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
150/150 100%No foreign tool servers
harness.mcp-hermeticNo call loaded a tool server the harness did not give it.
150/150 100%No silent fallbacks
harness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
150/150 100%Run completed
harness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
150/150 100%Structured replies arrived
harness.structured-outputEvery call that asked for structured output and ran to the end got it.
150/150 100%
A run of it
Plan-execute-verify on Return an expired, partially shipped order with Sonnet 5.5, repeat 1. It passed.
5 calls · 1m 20s · $0.177
- tool call
- denied by the guard
- tool returned an error
- shown no tools (reasoning only)
- lost its tools (server failed)
- failed
150 ranked runs of 150 recorded.
Provenance of the numbers on this page
- run set
- scaffold-or-scale-v1
- models
- Haiku 4.5, Sonnet 5.5, Opus 5.5
- effort
- high
- repeats
- 2
- fixtures
- replay
- runs
- 900 / 900
- strategy cost
- $354.44
- judge cost
- $9.19
- wall clock
- 9h 48m
- git
- fce5807b2d
Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.