Architecture
Gated-execute
Is review at the point of action worth a model call per action?
One agent acts, but every state-changing tool call must pass a read-only reviewer before it runs.
How it runs
1The executor
full toolset
One agent with the full toolset. Reads pass straight through; every call that would change state is held until a reviewer has ruled on it.
2The reviewer
read-only tools
A separate call per state-changing action. It holds the domain policy and the read-only tools, checks the current state and allows or denies. A denial returns to the executor as the tool’s result, with the reason, so it can adapt.
Where it can fail quietly
If a state-changing call can run without a review the gate is decoration, and an unreadable decision must count as a denial.
What the structure buys
Gated-execute against a single agent on the same tasks and repeats, on each model. The interval is on the paired difference, so it is as wide as the evidence is thin.
- Haiku 4.5=1 of 6
- Score
- 0.677
- 0.534–0.809
- Cost / run
- $0.862
- 4.8× single
- Ranked
- 50/50
not separable from single−0.094 [−0.215, +0.005]too few tasks carry the spread to show equivalence
- Sonnet 5.5=1 of 6
- Score
- 0.927
- 0.889–0.963
- Cost / run
- $0.495
- 2.6× single
- Ranked
- 50/50
not separable from single+0.028 [−0.012, +0.083]too few tasks carry the spread to show equivalence
- Opus 5.5=1 of 6
- Score
- 0.956
- 0.922–0.985
- Cost / run
- $0.934
- 2.5× single
- Ranked
- 50/50
not separable from single+0.016 [0.000, +0.041]too few tasks carry the spread to show equivalence
Task by task against a single agent
A mean can hide whether a design helps everywhere a little or on one task a lot. Each bar is the difference from a single agent on one task; the best-served tasks are at the top.
Where the money goes
A run of Gated-execute is 2 stages. This is what each one costs, so the price of the structure is separate from the price of the work.
- Haiku 4.5$0.862 per run · 50 runs
- Sonnet 5.5$0.495 per run · 50 runs
- Opus 5.5$0.934 per run · 50 runs
| Stage | Haiku 4.5 | Sonnet 5.5 | Opus 5.5 |
|---|---|---|---|
| The executor | $0.14216% | $0.11724% | $0.24626% |
| The reviewer | $0.72084% | $0.37876% | $0.68874% |
Did it run as designed?
Each check is recorded with every run. A run that fails one is excluded from every ranking and counted, never scored.
The design’s own checks
Every state change was reviewed
gated-execute.reviewedEvery call that changed the world had been allowed by a review before it ran, counted from the world’s own journal.
150/150 100%Every decision was read
gated-execute.decisionsEvery review that ran to its end gave its decision through the schema; an unreadable one is a denial.
150/150 100%Each call was shown the right tools
gated-execute.toolsThe executor was shown the task’s full toolset and each reviewer the read-only tools.
150/150 100%
The harness’s checksall 10 held in every run
No API failures
harness.apiNo model call failed at the API.
150/150 100%Builtin tools
harness.builtin-toolsA check recorded with the run.
150/150 100%Every call ended
harness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
150/150 100%No harness tool denied
harness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
150/150 100%Judge answered
harness.judgeAt least one grading call returned a usable verdict.
18/18 100%Tool servers connected
harness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
150/150 100%No foreign tool servers
harness.mcp-hermeticNo call loaded a tool server the harness did not give it.
150/150 100%No silent fallbacks
harness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
150/150 100%Run completed
harness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
150/150 100%Structured replies arrived
harness.structured-outputEvery call that asked for structured output and ran to the end got it.
150/150 100%
A run of it
Gated-execute on Cancel a fare the customer wrongly believes is refundable to their card with Haiku 4.5, repeat 1. It scored 0.000.
3 calls · 1m 17s · $0.086
- tool call
- denied by the guard
- tool returned an error
- shown no tools (reasoning only)
- lost its tools (server failed)
- failed
150 ranked runs of 150 recorded.
Provenance of the numbers on this page
- run set
- scaffold-or-scale-v1
- models
- Haiku 4.5, Sonnet 5.5, Opus 5.5
- effort
- high
- repeats
- 2
- fixtures
- replay
- runs
- 900 / 900
- strategy cost
- $354.44
- judge cost
- $9.19
- wall clock
- 9h 48m
- git
- fce5807b2d
Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.