research · web
Twelve HTTP status codes, by the book
Pre-registered hypothesis
Hypothesis
A tie on score: in a one-run calibration, single scored 1.0 with both Haiku 4.5 and Sonnet 5.5. Twelve independent lookups are what orchestrator-workers can split, so fan-out should show up as lower wall-clock time at a higher token cost. Grading is exact, so any score gap is not judge noise.
Written when the task was written, before any run. It is a prediction in words, so read it against what happened rather than trusting a verdict drawn for you.
What happened on Sonnet 5.5
- Highest mean score: Single 1.000
- Lowest: Gated-execute 1.000
Means over the runs that count; intervals and verdicts are on the results page.
Who solved it
On Sonnet 5.5, every architecture averages 1.000: this task does not separate them. The same task, every architecture, every model: one square per run.
Point at a square or tab to it to read its run. Each square opens its run.
The prompt every architecture was handed
For each HTTP status code below, report what the current HTTP Semantics specification says about it: its reason phrase, the section of that specification that defines it, and whether a response with that status code is heuristically cacheable (reusable by a cache without explicit freshness information). - 203 - 206 - 303 - 307 - 308 - 405 - 410 - 413 - 414 - 421 - 422 - 501 Research this with the provided web tools. Report, for every status code: - reason: the reason phrase exactly as the current specification gives it - section: the number of the section of the current specification that defines the code (just the number, e.g. "12.3.4") - heuristicallyCacheable: true if the specification defines responses with this code as heuristically cacheable, otherwise false - quote: a sentence from the numbered section of the specification that defines that status code (its own section, not an overview list or the table of contents), copied word for word (at least six words). - url: the URL of the page the quote comes from, which you opened with fetch_page. Write whatever explanation you like, then end the answer with one fenced JSON block of exactly this shape, with one entry per status code: ```json {"answers":[{"code":"203","reason":"…","section":"…","heuristicallyCacheable":true,"quote":"<a sentence copied word for word>","url":"<the URL of that page, as you opened it>"}]} ``` Quotes are checked against the text of the page at their URL, so copy them exactly. The block must be valid JSON: write a double quote inside a string as \".
The domain policy
Appended to the system prompt for every architecture, so none of them gets a rules advantage.
Research rules: - The `search` and `fetch_page` tools are your only access to the web. Any other browsing tool is unavailable. - Search snippets are previews, not sources. Open a page with `fetch_page` before relying on what it says. - Long pages arrive one window at a time. Continue with `offset`, or use `find` to jump to a phrase. - Cite the URL of every source you rely on, next to the claim it supports. - Never cite a URL you did not open with `fetch_page`. A citation to an unopened page is scored as a fabrication. - Quotes must be copied word for word from the page you cite; they are checked against the page text. - Quote only passages the tools have shown you in this session. A quote recalled from memory does not count, even when the page contains it. - When sources disagree, say so explicitly and attribute each position, instead of silently picking one.
The traps it sets
A trap is a specific way to go wrong, declared before any run and tied to the assertions or rules that register it. A trap fired in a run when one of those failed. On Sonnet 5.5:
Where it was lost
On Sonnet 5.5, no architecture fell short of any graded item. A score says an architecture lost points. This says which ones: for each graded item, the runs that earned less than full credit, out of the runs that count.
In numbers, on Claude Sonnet 5.5
- Gated-execute1.000
2 runs100% pass$0.109 / run23.4s
- Orchestrator-workers1.000
2 runs100% pass$0.340 / run53.7s
- Plan-execute1.000
2 runs100% pass$0.151 / run38.7s
- Plan-execute-verify1.000
2 runs100% pass$0.255 / run59.9s
- Propose-select1.000
2 runs100% pass$0.507 / run59.4s
- Single1.000
2 runs100% pass$0.119 / run27.0s
| Strategy | n | Mean score | Pass | Cost / run | Latency | Failure modes |
|---|---|---|---|---|---|---|
| Gated-execute | 2 | 1.000 | 100% | $0.109 | 23.4s | — |
| Orchestrator-workers | 2 | 1.000 | 100% | $0.340 | 53.7s | — |
| Plan-execute | 2 | 1.000 | 100% | $0.151 | 38.7s | — |
| Plan-execute-verify | 2 | 1.000 | 100% | $0.255 | 59.9s | — |
| Propose-select | 2 | 1.000 | 100% | $0.507 | 59.4s | — |
| Single | 2 | 1.000 | 100% | $0.119 | 27.0s | — |
Every run on Sonnet 5.512 runs · $2.96 spent
Provenance of the numbers on this page
- run set
- scaffold-or-scale-v1
- models
- Haiku 4.5, Sonnet 5.5, Opus 5.5
- effort
- high
- repeats
- 2
- fixtures
- replay
- runs
- 900 / 900
- strategy cost
- $354.44
- judge cost
- $9.19
- wall clock
- 9h 48m
- git
- fce5807b2d
Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.