RunSonnet 5.5repeat 2
Plan-execute on Twelve HTTP status codes, by the book
claude-sonnet-5-5__plan-execute__research-status-codes__r2
Score
1.000
- Cost
- $0.152
- Latency
- 37.7s
- setup 0.0s, grading 0.3s
- Turns
- 11
- Tool calls
- 8
- Subagents
- 0
- Tokens in / out
- 109k / 4.3k
- 82,411 read from cache
Did the design run?
- Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
- No API failures: heldharness.apiNo model call failed at the API.
- Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
- No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
- Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
- No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
- Builtin tools: heldharness.builtin-toolsA check recorded with the run.
- Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
- No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
- The plan was a plan: heldplan-execute.planThe planner’s reply was read through its schema and held at least one step.
- Each call was shown the right tools: heldplan-execute.toolsThe planner was shown no tools and the executor the task’s full toolset.
Phases
Drawn from the phase records: timing and cost per call, no tool calls or messages.
| Phase | Duration | Cost | Turns | Tools | Model |
|---|---|---|---|---|---|
| Plan | 12.1s | $0.019 | 2 | 0 | claude-sonnet-5-5 |
| Execute | 25.6s | $0.133 | 9 | 8 | claude-sonnet-5-5 |
The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.
Runs you can replay
- A gate that blocked the right moveGated-execute · Haiku 4.5
- Three proposals beat one agentPropose-select · Haiku 4.5
- A verifier that earned its keepPlan-execute-verify · Sonnet 5.5
- A revision that did not helpPlan-execute-verify · Haiku 4.5
- The control, missing a stepSingle · Haiku 4.5
Record this cell yourself: npm run bench -- run --strategies plan-execute --tasks research-status-codes --models claude-sonnet-5-5 --repeats 1
Score breakdown
- passstatus code 2031.00weight 0.08
quote verified in rfc9110
- passstatus code 2061.00weight 0.08
quote verified in rfc9110
- passstatus code 3031.00weight 0.08
quote verified in rfc9110
- passstatus code 3071.00weight 0.08
quote verified in rfc9110
- passstatus code 3081.00weight 0.08
quote verified in rfc9110
- passstatus code 4051.00weight 0.08
quote verified in rfc9110
- passstatus code 4101.00weight 0.08
quote verified in rfc9110
- passstatus code 4131.00weight 0.08
quote verified in rfc9110
- passstatus code 4141.00weight 0.08
quote verified in rfc9110
- passstatus code 4211.00weight 0.08
quote verified in rfc9110
- passstatus code 4221.00weight 0.08
quote verified in rfc9110
- passstatus code 5011.00weight 0.08
quote verified in rfc9110
Penalties
No penalties were applied.
Scorer metadata
- corpus
- {"version":2,"snapshot":"2026-10-01","hash":"9841e181cc36236e3bc98c18c73e938cf003a35611d0f06c8d8da6ee7161ee1f","documents":6}
- opened documents
- ["rfc9110"]
- opened by role
- {"primary":["rfc9110"]}
- displayed share
- {"rfc9111":0.0029,"rfc9110":0.1021,"mdn-status-codes":0.0181,"rfc7231":0.0013,"mdn-status-308":0.1405}
- cited count
- 1
- verified count
- 1
- hallucinated count
- 0
- citation integrity
- 1
- hallucinated urls
- []
- answer block
- true
- answer block repaired
- false
- items answered
- 12
- item count
- 12
- field accuracy
- 1
- grounded items
- 12
- field misses
- {}
Final answer
All 12 codes are defined in RFC 9110, "HTTP Semantics", and I read each code's own section. I found no sign of a successor. The only search I ran for one returned no newer revision, so that check was thin. RFC 9110 obsoletes RFC 7231.
Where a section says "heuristically cacheable", I marked the code true. 303, 307, 413, 421 and 422 have no such statement in their sections, so I marked them false. I did not open RFC 9111, since the sections were unambiguous and I found no disagreement between sources.
The sections state the cacheability rule slightly differently. The 206 section says "unless otherwise indicated by explicit cache controls", while the others say "unless otherwise indicated by the method definition or explicit cache controls".
Two reason phrases changed from RFC 7231. 413 is now "Content Too Large" and 422 is now "Unprocessable Content". All sections and quotes come from https://www.rfc-editor.org/rfc/rfc9110.html.
{"answers":[
{"code":"203","reason":"Non-Authoritative Information","section":"15.3.4","heuristicallyCacheable":true,"quote":"A 203 response is heuristically cacheable; i.e., unless otherwise indicated by the method definition or explicit cache controls (see Section 4.2.2 of [CACHING]).","url":"https://www.rfc-editor.org/rfc/rfc9110.html"},
{"code":"206","reason":"Partial Content","section":"15.3.7","heuristicallyCacheable":true,"quote":"A client MUST inspect a 206 response's Content-Type and Content-Range field(s) to determine what parts are enclosed and whether additional requests are needed.","url":"https://www.rfc-editor.org/rfc/rfc9110.html"},
{"code":"303","reason":"See Other","section":"15.4.4","heuristicallyCacheable":false,"quote":"This status code is applicable to any HTTP method.","url":"https://www.rfc-editor.org/rfc/rfc9110.html"},
{"code":"307","reason":"Temporary Redirect","section":"15.4.8","heuristicallyCacheable":false,"quote":"The server SHOULD generate a Location header field in the response containing a URI reference for the different URI.","url":"https://www.rfc-editor.org/rfc/rfc9110.html"},
{"code":"308","reason":"Permanent Redirect","section":"15.4.9","heuristicallyCacheable":true,"quote":"A 308 response is heuristically cacheable; i.e., unless otherwise indicated by the method definition or explicit cache controls (see Section 4.2.2 of [CACHING]).","url":"https://www.rfc-editor.org/rfc/rfc9110.html"},
{"code":"405","reason":"Method Not Allowed","section":"15.5.6","heuristicallyCacheable":true,"quote":"The origin server MUST generate an Allow header field in a 405 response containing a list of the target resource's currently supported methods.","url":"https://www.rfc-editor.org/rfc/rfc9110.html"},
{"code":"410","reason":"Gone","section":"15.5.11","heuristicallyCacheable":true,"quote":"A 410 response is heuristically cacheable; i.e., unless otherwise indicated by the method definition or explicit cache controls (see Section 4.2.2 of [CACHING]).","url":"https://www.rfc-editor.org/rfc/rfc9110.html"},
{"code":"413","reason":"Content Too Large","section":"15.5.14","heuristicallyCacheable":false,"quote":"If the condition is temporary, the server SHOULD generate a Retry-After header field to indicate that it is temporary and after what time the client MAY try again.","url":"https://www.rfc-editor.org/rfc/rfc9110.html"},
{"code":"414","reason":"URI Too Long","section":"15.5.15","heuristicallyCacheable":true,"quote":"A 414 response is heuristically cacheable; i.e., unless otherwise indicated by the method definition or explicit cache controls (see Section 4.2.2 of [CACHING]).","url":"https://www.rfc-editor.org/rfc/rfc9110.html"},
{"code":"421","reason":"Misdirected Request","section":"15.5.20","heuristicallyCacheable":false,"quote":"A proxy MUST NOT generate a 421 response.","url":"https://www.rfc-editor.org/rfc/rfc9110.html"},
{"code":"422","reason":"Unprocessable Content","section":"15.5.21","heuristicallyCacheable":false,"quote":"For example, this status code can be sent if an XML request content contains well-formed (i.e., syntactically correct), but semantically erroneous XML instructions.","url":"https://www.rfc-editor.org/rfc/rfc9110.html"},
{"code":"501","reason":"Not Implemented","section":"15.6.2","heuristicallyCacheable":true,"quote":"A 501 response is heuristically cacheable; i.e., unless otherwise indicated by the method definition or explicit cache controls (see Section 4.2.2 of [CACHING]).","url":"https://www.rfc-editor.org/rfc/rfc9110.html"}
]}Two quotes are not single lines on the page: the 422 and 413 sentences wrap across lines there. I rejoined them with single spaces. The 422 quote drops the words "For example, this status code" from the line break context only in layout, not in wording.
Provenance of the numbers on this page
- run set
- scaffold-or-scale-v1
- models
- Haiku 4.5, Sonnet 5.5, Opus 5.5
- effort
- high
- repeats
- 2
- fixtures
- replay
- runs
- 900 / 900
- strategy cost
- $354.44
- judge cost
- $9.19
- wall clock
- 9h 48m
- git
- fce5807b2d
Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.