ScaffoldBench

research · web

Twelve HTTP status codes, by the book

mediumhttpspecificationsbreadthgold-key
Task id
research-status-codes
Turn limit
40
Runs on Haiku 4.5
12
Pass rate
100%

Pre-registered hypothesis

Hypothesis

A tie on score: in a one-run calibration, single scored 1.0 with both Haiku 4.5 and Sonnet 5.5. Twelve independent lookups are what orchestrator-workers can split, so fan-out should show up as lower wall-clock time at a higher token cost. Grading is exact, so any score gap is not judge noise.

Written when the task was written, before any run. It is a prediction in words, so read it against what happened rather than trusting a verdict drawn for you.

What happened on Haiku 4.5

  • Highest mean score: Single 1.000
  • Lowest: Gated-execute 1.000

Means over the runs that count; intervals and verdicts are on the results page.

Who solved it

On Haiku 4.5, every architecture averages 1.000: this task does not separate them. The same task, every architecture, every model: one square per run.

1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
0.917 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
0.875 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count

Point at a square or tab to it to read its run. Each square opens its run.

passedpartial creditfailedexcluded: integrity check failedexcluded: API errorOne square per run; each opens its run.

The prompt every architecture was handed

For each HTTP status code below, report what the current HTTP Semantics specification says about it: its reason phrase, the section of that specification that defines it, and whether a response with that status code is heuristically cacheable (reusable by a cache without explicit freshness information). - 203 - 206 - 303 - 307 - 308 - 405 - 410 - 413 - 414 - 421 - 422 - 501 Research this with the provided web tools. Report, for every status code: - reason: the reason phrase exactly as the current specification gives it - section: the number of the section of the current specification that defines the code (just the number, e.g. "12.3.4") - heuristicallyCacheable: true if the specification defines responses with this code as heuristically cacheable, otherwise false - quote: a sentence from the numbered section of the specification that defines that status code (its own section, not an overview list or the table of contents), copied word for word (at least six words). - url: the URL of the page the quote comes from, which you opened with fetch_page. Write whatever explanation you like, then end the answer with one fenced JSON block of exactly this shape, with one entry per status code: ```json {"answers":[{"code":"203","reason":"…","section":"…","heuristicallyCacheable":true,"quote":"<a sentence copied word for word>","url":"<the URL of that page, as you opened it>"}]} ``` Quotes are checked against the text of the page at their URL, so copy them exactly. The block must be valid JSON: write a double quote inside a string as \".

The domain policy

Appended to the system prompt for every architecture, so none of them gets a rules advantage.

Research rules: - The `search` and `fetch_page` tools are your only access to the web. Any other browsing tool is unavailable. - Search snippets are previews, not sources. Open a page with `fetch_page` before relying on what it says. - Long pages arrive one window at a time. Continue with `offset`, or use `find` to jump to a phrase. - Cite the URL of every source you rely on, next to the claim it supports. - Never cite a URL you did not open with `fetch_page`. A citation to an unopened page is scored as a fabrication. - Quotes must be copied word for word from the page you cite; they are checked against the page text. - Quote only passages the tools have shown you in this session. A quote recalled from memory does not count, even when the page contains it. - When sources disagree, say so explicitly and attribute each position, instead of silently picking one.

The traps it sets

A trap is a specific way to go wrong, declared before any run and tied to the assertions or rules that register it. A trap fired in a run when one of those failed. On Haiku 4.5:

Twelve HTTP status codes, by the book3 traps · runs that fell in, of runs ranked
Trap
Single
Plan-execute
Orchestrator-workers
Plan-execute-verify
Propose-select
Gated-execute
renamed-reason-phrasesRFC 9110 renamed 413 ("Payload Too Large" in RFC 7231) to "Content Too Large" and 422 ("Unprocessable Entity" in WebDAV) to "Unprocessable Content". Memory and the obsolete RFC give the old names.
Single fell into renamed-reason-phrases in 0 of 2 runs
Plan-execute fell into renamed-reason-phrases in 0 of 2 runs
Orchestrator-workers fell into renamed-reason-phrases in 0 of 2 runs
Plan-execute-verify fell into renamed-reason-phrases in 0 of 2 runs
Propose-select fell into renamed-reason-phrases in 0 of 2 runs
Gated-execute fell into renamed-reason-phrases in 0 of 2 runs
obsolete-sectionsRFC 7231 is in the corpus and numbers its status codes 6.x; the current sections are 15.x.
Single fell into obsolete-sections in 0 of 2 runs
Plan-execute fell into obsolete-sections in 0 of 2 runs
Orchestrator-workers fell into obsolete-sections in 0 of 2 runs
Plan-execute-verify fell into obsolete-sections in 0 of 2 runs
Propose-select fell into obsolete-sections in 0 of 2 runs
Gated-execute fell into obsolete-sections in 0 of 2 runs
secondary-quoteQuoting MDN, or the overview list, instead of the section that defines the code.
Single fell into secondary-quote in 0 of 2 runs
Plan-execute fell into secondary-quote in 0 of 2 runs
Orchestrator-workers fell into secondary-quote in 0 of 2 runs
Plan-execute-verify fell into secondary-quote in 0 of 2 runs
Propose-select fell into secondary-quote in 0 of 2 runs
Gated-execute fell into secondary-quote in 0 of 2 runs

Where it was lost

On Haiku 4.5, no architecture fell short of any graded item. A score says an architecture lost points. This says which ones: for each graded item, the runs that earned less than full credit, out of the runs that count.

Graded items on Haiku 4.5runs that fell short, of runs that count
Graded item
Single
Plan-execute
Orchestrator-workers
Plan-execute-verify
Propose-select
Gated-execute
status code 203weight 1
Single fell short of "status code 203" in 0 of 2 runs
Plan-execute fell short of "status code 203" in 0 of 2 runs
Orchestrator-workers fell short of "status code 203" in 0 of 2 runs
Plan-execute-verify fell short of "status code 203" in 0 of 2 runs
Propose-select fell short of "status code 203" in 0 of 2 runs
Gated-execute fell short of "status code 203" in 0 of 2 runs
status code 206weight 1
Single fell short of "status code 206" in 0 of 2 runs
Plan-execute fell short of "status code 206" in 0 of 2 runs
Orchestrator-workers fell short of "status code 206" in 0 of 2 runs
Plan-execute-verify fell short of "status code 206" in 0 of 2 runs
Propose-select fell short of "status code 206" in 0 of 2 runs
Gated-execute fell short of "status code 206" in 0 of 2 runs
status code 303weight 1
Single fell short of "status code 303" in 0 of 2 runs
Plan-execute fell short of "status code 303" in 0 of 2 runs
Orchestrator-workers fell short of "status code 303" in 0 of 2 runs
Plan-execute-verify fell short of "status code 303" in 0 of 2 runs
Propose-select fell short of "status code 303" in 0 of 2 runs
Gated-execute fell short of "status code 303" in 0 of 2 runs
status code 307weight 1
Single fell short of "status code 307" in 0 of 2 runs
Plan-execute fell short of "status code 307" in 0 of 2 runs
Orchestrator-workers fell short of "status code 307" in 0 of 2 runs
Plan-execute-verify fell short of "status code 307" in 0 of 2 runs
Propose-select fell short of "status code 307" in 0 of 2 runs
Gated-execute fell short of "status code 307" in 0 of 2 runs
status code 308weight 1
Single fell short of "status code 308" in 0 of 2 runs
Plan-execute fell short of "status code 308" in 0 of 2 runs
Orchestrator-workers fell short of "status code 308" in 0 of 2 runs
Plan-execute-verify fell short of "status code 308" in 0 of 2 runs
Propose-select fell short of "status code 308" in 0 of 2 runs
Gated-execute fell short of "status code 308" in 0 of 2 runs
status code 405weight 1
Single fell short of "status code 405" in 0 of 2 runs
Plan-execute fell short of "status code 405" in 0 of 2 runs
Orchestrator-workers fell short of "status code 405" in 0 of 2 runs
Plan-execute-verify fell short of "status code 405" in 0 of 2 runs
Propose-select fell short of "status code 405" in 0 of 2 runs
Gated-execute fell short of "status code 405" in 0 of 2 runs
status code 410weight 1
Single fell short of "status code 410" in 0 of 2 runs
Plan-execute fell short of "status code 410" in 0 of 2 runs
Orchestrator-workers fell short of "status code 410" in 0 of 2 runs
Plan-execute-verify fell short of "status code 410" in 0 of 2 runs
Propose-select fell short of "status code 410" in 0 of 2 runs
Gated-execute fell short of "status code 410" in 0 of 2 runs
status code 413weight 1
Single fell short of "status code 413" in 0 of 2 runs
Plan-execute fell short of "status code 413" in 0 of 2 runs
Orchestrator-workers fell short of "status code 413" in 0 of 2 runs
Plan-execute-verify fell short of "status code 413" in 0 of 2 runs
Propose-select fell short of "status code 413" in 0 of 2 runs
Gated-execute fell short of "status code 413" in 0 of 2 runs
status code 414weight 1
Single fell short of "status code 414" in 0 of 2 runs
Plan-execute fell short of "status code 414" in 0 of 2 runs
Orchestrator-workers fell short of "status code 414" in 0 of 2 runs
Plan-execute-verify fell short of "status code 414" in 0 of 2 runs
Propose-select fell short of "status code 414" in 0 of 2 runs
Gated-execute fell short of "status code 414" in 0 of 2 runs
status code 421weight 1
Single fell short of "status code 421" in 0 of 2 runs
Plan-execute fell short of "status code 421" in 0 of 2 runs
Orchestrator-workers fell short of "status code 421" in 0 of 2 runs
Plan-execute-verify fell short of "status code 421" in 0 of 2 runs
Propose-select fell short of "status code 421" in 0 of 2 runs
Gated-execute fell short of "status code 421" in 0 of 2 runs
status code 422weight 1
Single fell short of "status code 422" in 0 of 2 runs
Plan-execute fell short of "status code 422" in 0 of 2 runs
Orchestrator-workers fell short of "status code 422" in 0 of 2 runs
Plan-execute-verify fell short of "status code 422" in 0 of 2 runs
Propose-select fell short of "status code 422" in 0 of 2 runs
Gated-execute fell short of "status code 422" in 0 of 2 runs
status code 501weight 1
Single fell short of "status code 501" in 0 of 2 runs
Plan-execute fell short of "status code 501" in 0 of 2 runs
Orchestrator-workers fell short of "status code 501" in 0 of 2 runs
Plan-execute-verify fell short of "status code 501" in 0 of 2 runs
Propose-select fell short of "status code 501" in 0 of 2 runs
Gated-execute fell short of "status code 501" in 0 of 2 runs

In numbers, on Claude Haiku 4.5

  • Gated-execute
    1.000

    2 runs100% pass$0.109 / run58.0s

  • Orchestrator-workers
    1.000

    2 runs100% pass$0.232 / run1m 19s

  • Plan-execute
    1.000

    2 runs100% pass$0.167 / run2m 11s

  • Plan-execute-verify
    1.000

    2 runs100% pass$0.289 / run3m 17s

  • Propose-select
    1.000

    2 runs100% pass$0.420 / run2m 17s

  • Single
    1.000

    2 runs100% pass$0.121 / run1m 14s

Every run on Haiku 4.512 runs · $2.68 spent

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.