research · web
Where ten HTTP header fields are defined
Pre-registered hypothesis
Hypothesis
The strongest case for fan-out, ten lookups across four current RFCs, but expect a tie on score: in a one-run calibration, single scored 1.0 with both Haiku 4.5 and Sonnet 5.5 once it read the corpus (closed-book, Haiku named an obsolete RFC for 8 of 10 fields). Fan-out should cut wall-clock time at a higher token cost.
Written when the task was written, before any run. It is a prediction in words, so read it against what happened rather than trusting a verdict drawn for you.
What happened on Haiku 4.5
- Highest mean score: Single 1.000
- Lowest: Plan-execute 0.619
Means over the runs that count; intervals and verdicts are on the results page.
Who solved it
On Haiku 4.5, Single, Plan-execute-verify, Propose-select and Gated-execute score highest at 1.000 and Plan-execute lowest at 0.619. The same task, every architecture, every model: one square per run.
Point at a square or tab to it to read its run. Each square opens its run.
The prompt every architecture was handed
For each HTTP header field below, find the RFC that currently defines it and the section of that RFC that contains its definition. An RFC that has since been obsoleted does not count. - Age - Cache-Control - Expires - Last-Modified - ETag - Location - Retry-After - Vary - Strict-Transport-Security - Link Research this with the provided web tools. Report, for every header field: - rfc: the number of the RFC that currently defines the field - section: the number of the section of that RFC that defines the field (just the number, e.g. "12.3.4") - quote: a sentence from the section of that RFC that defines the field, copied word for word (at least six words). - url: the URL of the page the quote comes from, which you opened with fetch_page. Write whatever explanation you like, then end the answer with one fenced JSON block of exactly this shape, with one entry per header field: ```json {"answers":[{"field":"Age","rfc":"…","section":"…","quote":"<a sentence copied word for word>","url":"<the URL of that page, as you opened it>"}]} ``` Quotes are checked against the text of the page at their URL, so copy them exactly. The block must be valid JSON: write a double quote inside a string as \".
The domain policy
Appended to the system prompt for every architecture, so none of them gets a rules advantage.
Research rules: - The `search` and `fetch_page` tools are your only access to the web. Any other browsing tool is unavailable. - Search snippets are previews, not sources. Open a page with `fetch_page` before relying on what it says. - Long pages arrive one window at a time. Continue with `offset`, or use `find` to jump to a phrase. - Cite the URL of every source you rely on, next to the claim it supports. - Never cite a URL you did not open with `fetch_page`. A citation to an unopened page is scored as a fabrication. - Quotes must be copied word for word from the page you cite; they are checked against the page text. - Quote only passages the tools have shown you in this session. A quote recalled from memory does not count, even when the page contains it. - When sources disagree, say so explicitly and attribute each position, instead of silently picking one.
The traps it sets
A trap is a specific way to go wrong, declared before any run and tied to the assertions or rules that register it. A trap fired in a run when one of those failed. On Haiku 4.5:
Where it was lost
On Haiku 4.5, the item most often missed is “header field Last-Modified”, by 17% of runs. A score says an architecture lost points. This says which ones: for each graded item, the runs that earned less than full credit, out of the runs that count.
How the failures failed2wrong answer
In numbers, on Claude Haiku 4.5
- Gated-execute1.000
2 runs100% pass$0.179 / run1m 29s
- Plan-execute-verify1.000
2 runs100% pass$0.355 / run3m 5s
- Propose-select1.000
2 runs100% pass$0.602 / run2m 14s
- Single1.000
2 runs100% pass$0.203 / run1m 22s
- Orchestrator-workers0.950
2 runs50% pass$0.251 / run1m 2s
wrong answer ×1
- Plan-execute0.619
2 runs50% pass$0.182 / run1m 46s
wrong answer ×1
| Strategy | n | Mean score | Pass | Cost / run | Latency | Failure modes |
|---|---|---|---|---|---|---|
| Gated-execute | 2 | 1.000 | 100% | $0.179 | 1m 29s | — |
| Plan-execute-verify | 2 | 1.000 | 100% | $0.355 | 3m 5s | — |
| Propose-select | 2 | 1.000 | 100% | $0.602 | 2m 14s | — |
| Single | 2 | 1.000 | 100% | $0.203 | 1m 22s | — |
| Orchestrator-workers | 2 | 0.950 | 50% | $0.251 | 1m 2s | wrong answer ×1 |
| Plan-execute | 2 | 0.619 | 50% | $0.182 | 1m 46s | wrong answer ×1 |
Every run on Haiku 4.512 runs · $3.54 spent
Provenance of the numbers on this page
- run set
- scaffold-or-scale-v1
- models
- Haiku 4.5, Sonnet 5.5, Opus 5.5
- effort
- high
- repeats
- 2
- fixtures
- replay
- runs
- 900 / 900
- strategy cost
- $354.44
- judge cost
- $9.19
- wall clock
- 9h 48m
- git
- fce5807b2d
Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.