ScaffoldBench

research · web

HTTP 301 vs 308

easyhttpspecificationscomparison
Task id
research-http-redirects
Turn limit
22
Runs on Haiku 4.5
12
Pass rate
0%

Pre-registered hypothesis

Hypothesis

Near the ceiling for every architecture: all five claims sit in two short sections of RFC 9110. The one place a verify phase can earn its cost is the MDN trap below.

Written when the task was written, before any run. It is a prediction in words, so read it against what happened rather than trusting a verdict drawn for you.

What happened on Haiku 4.5

  • Highest mean score: Orchestrator-workers 0.917
  • A single agent: 0.885
  • Lowest: Propose-select 0.573

Means over the runs that count; intervals and verdicts are on the results page.

Who solved it

On Haiku 4.5, Orchestrator-workers and Gated-execute score highest at 0.917 and Propose-select lowest at 0.573. The same task, every architecture, every model: one square per run.

Haiku 4.5
Sonnet 5.5
Opus 5.5
Single
0.885 mean score of the runs that count
0.938 mean score of the runs that count
1.000 mean score of the runs that count
Plan-execute
0.885 mean score of the runs that count
0.938 mean score of the runs that count
1.000 mean score of the runs that count
Orchestrator-workers
0.917 mean score of the runs that count
1.000 mean score of the runs that count
0.896 mean score of the runs that count
Plan-execute-verify
0.885 mean score of the runs that count
0.938 mean score of the runs that count
1.000 mean score of the runs that count
Propose-select
0.573 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
Gated-execute
0.917 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count

Point at a square or tab to it to read its run. Each square opens its run.

passedpartial creditfailedexcluded: integrity check failedexcluded: API errorOne square per run; each opens its run.

The prompt every architecture was handed

What is the difference between HTTP status codes 301 and 308? Explain what each one means, how a client is expected to treat the request method and body when following each, whether each is cacheable, and where each is currently specified. Research this with the provided web tools and write a complete, self-contained answer. Write the answer for the user in prose. Then list the evidence for your key claims: end the answer with one fenced JSON block of exactly this shape, one entry per claim: ```json {"evidence": [{"claim": "<the claim, in your words>", "quote": "<a sentence copied word for word from the page>", "url": "<the URL of that page, as you opened it>"}]} ``` Every quote is checked against the text of the page at its URL. A paraphrase, or a quote from a page you did not open with fetch_page, does not count as evidence. The block must be valid JSON: write a double quote inside a string as \".

The domain policy

Appended to the system prompt for every architecture, so none of them gets a rules advantage.

Research rules: - The `search` and `fetch_page` tools are your only access to the web. Any other browsing tool is unavailable. - Search snippets are previews, not sources. Open a page with `fetch_page` before relying on what it says. - Long pages arrive one window at a time. Continue with `offset`, or use `find` to jump to a phrase. - Cite the URL of every source you rely on, next to the claim it supports. - Never cite a URL you did not open with `fetch_page`. A citation to an unopened page is scored as a fabrication. - Quotes must be copied word for word from the page you cite; they are checked against the page text. - Quote only passages the tools have shown you in this session. A quote recalled from memory does not count, even when the page contains it. - When sources disagree, say so explicitly and attribute each position, instead of silently picking one.

The traps it sets

A trap is a specific way to go wrong, declared before any run and tied to the assertions or rules that register it. A trap fired in a run when one of those failed. On Haiku 4.5:

HTTP 301 vs 3083 traps · runs that fell in, of runs ranked
Trap
Single
Plan-execute
Orchestrator-workers
Plan-execute-verify
Propose-select
Gated-execute
mdn-301-methodMDN's 308 page says a 301 'requires the request method and the body to remain unchanged'; RFC 9110 says a user agent MAY change POST to GET. Repeating MDN contradicts the primary source.
Single fell into mdn-301-method in 0 of 2 runs
Plan-execute fell into mdn-301-method in 0 of 2 runs
Orchestrator-workers fell into mdn-301-method in 0 of 2 runs
Plan-execute-verify fell into mdn-301-method in 0 of 2 runs
Propose-select fell into mdn-301-method in 0 of 2 runs
Gated-execute fell into mdn-301-method in 0 of 2 runs
rfc7231-cacheable-listRFC 7231's list of cacheable-by-default codes predates 308; an answer built on it gets 308 wrong.
Single fell into rfc7231-cacheable-list in 0 of 2 runs
Plan-execute fell into rfc7231-cacheable-list in 0 of 2 runs
Orchestrator-workers fell into rfc7231-cacheable-list in 0 of 2 runs
Plan-execute-verify fell into rfc7231-cacheable-list in 0 of 2 runs
Propose-select fell into rfc7231-cacheable-list in 0 of 2 runs
Gated-execute fell into rfc7231-cacheable-list in 0 of 2 runs
rfc7538-currentCiting RFC 7538 as the current definition of 308; RFC 9110 obsoletes it.
Single fell into rfc7538-current in 0 of 2 runs
Plan-execute fell into rfc7538-current in 0 of 2 runs
Orchestrator-workers fell into rfc7538-current in 0 of 2 runs
Plan-execute-verify fell into rfc7538-current in 0 of 2 runs
Propose-select fell into rfc7538-current in 0 of 2 runs
Gated-execute fell into rfc7538-current in 0 of 2 runs

Where it was lost

On Haiku 4.5, the item most often missed is “States that both codes are currently specified in RFC 9110 (HTTP Semantics), and that 308 was originally defined in RFC 7538, which RFC 9110 obsoletes.”, by 92% of runs. A score says an architecture lost points. This says which ones: for each graded item, the runs that earned less than full credit, out of the runs that count.

Graded items on Haiku 4.5runs that fell short, of runs that count
Graded item
Single
Plan-execute
Orchestrator-workers
Plan-execute-verify
Propose-select
Gated-execute
States that 301 is "Moved Permanently" and 308 is "Permanent Redirect", and that both indicate the target resource has been assigned a new permanent URI.weight 1
Single fell short of "States that 301 is "Moved Permanently" and 308 is "Permanent Redirect", and that both indicate the target resource has been assigned a new permanent URI." in 0 of 2 runs
Plan-execute fell short of "States that 301 is "Moved Permanently" and 308 is "Permanent Redirect", and that both indicate the target resource has been assigned a new permanent URI." in 0 of 2 runs
Orchestrator-workers fell short of "States that 301 is "Moved Permanently" and 308 is "Permanent Redirect", and that both indicate the target resource has been assigned a new permanent URI." in 0 of 2 runs
Plan-execute-verify fell short of "States that 301 is "Moved Permanently" and 308 is "Permanent Redirect", and that both indicate the target resource has been assigned a new permanent URI." in 0 of 2 runs
Propose-select fell short of "States that 301 is "Moved Permanently" and 308 is "Permanent Redirect", and that both indicate the target resource has been assigned a new permanent URI." in 1 of 2 runs
Gated-execute fell short of "States that 301 is "Moved Permanently" and 308 is "Permanent Redirect", and that both indicate the target resource has been assigned a new permanent URI." in 1 of 2 runs
States that a client following a 308 must not change the request method: a POST is repeated as a POST, with its body.weight 1.5
Single fell short of "States that a client following a 308 must not change the request method: a POST is repeated as a POST, with its body." in 0 of 2 runs
Plan-execute fell short of "States that a client following a 308 must not change the request method: a POST is repeated as a POST, with its body." in 0 of 2 runs
Orchestrator-workers fell short of "States that a client following a 308 must not change the request method: a POST is repeated as a POST, with its body." in 0 of 2 runs
Plan-execute-verify fell short of "States that a client following a 308 must not change the request method: a POST is repeated as a POST, with its body." in 0 of 2 runs
Propose-select fell short of "States that a client following a 308 must not change the request method: a POST is repeated as a POST, with its body." in 1 of 2 runs
Gated-execute fell short of "States that a client following a 308 must not change the request method: a POST is repeated as a POST, with its body." in 0 of 2 runs
States that when following a 301 a user agent MAY change the request method from POST to GET (for historical reasons), which is the behaviour 308 exists to rule out.weight 1.5
Single fell short of "States that when following a 301 a user agent MAY change the request method from POST to GET (for historical reasons), which is the behaviour 308 exists to rule out." in 0 of 2 runs
Plan-execute fell short of "States that when following a 301 a user agent MAY change the request method from POST to GET (for historical reasons), which is the behaviour 308 exists to rule out." in 0 of 2 runs
Orchestrator-workers fell short of "States that when following a 301 a user agent MAY change the request method from POST to GET (for historical reasons), which is the behaviour 308 exists to rule out." in 0 of 2 runs
Plan-execute-verify fell short of "States that when following a 301 a user agent MAY change the request method from POST to GET (for historical reasons), which is the behaviour 308 exists to rule out." in 0 of 2 runs
Propose-select fell short of "States that when following a 301 a user agent MAY change the request method from POST to GET (for historical reasons), which is the behaviour 308 exists to rule out." in 1 of 2 runs
Gated-execute fell short of "States that when following a 301 a user agent MAY change the request method from POST to GET (for historical reasons), which is the behaviour 308 exists to rule out." in 0 of 2 runs
States that both 301 and 308 responses are heuristically cacheable, i.e. a cache may reuse them by default unless the method definition or explicit cache controls say otherwise.weight 1
Single fell short of "States that both 301 and 308 responses are heuristically cacheable, i.e. a cache may reuse them by default unless the method definition or explicit cache controls say otherwise." in 0 of 2 runs
Plan-execute fell short of "States that both 301 and 308 responses are heuristically cacheable, i.e. a cache may reuse them by default unless the method definition or explicit cache controls say otherwise." in 0 of 2 runs
Orchestrator-workers fell short of "States that both 301 and 308 responses are heuristically cacheable, i.e. a cache may reuse them by default unless the method definition or explicit cache controls say otherwise." in 0 of 2 runs
Plan-execute-verify fell short of "States that both 301 and 308 responses are heuristically cacheable, i.e. a cache may reuse them by default unless the method definition or explicit cache controls say otherwise." in 0 of 2 runs
Propose-select fell short of "States that both 301 and 308 responses are heuristically cacheable, i.e. a cache may reuse them by default unless the method definition or explicit cache controls say otherwise." in 1 of 2 runs
Gated-execute fell short of "States that both 301 and 308 responses are heuristically cacheable, i.e. a cache may reuse them by default unless the method definition or explicit cache controls say otherwise." in 0 of 2 runs
States that both codes are currently specified in RFC 9110 (HTTP Semantics), and that 308 was originally defined in RFC 7538, which RFC 9110 obsoletes.weight 1
Single fell short of "States that both codes are currently specified in RFC 9110 (HTTP Semantics), and that 308 was originally defined in RFC 7538, which RFC 9110 obsoletes." in 2 of 2 runs
Plan-execute fell short of "States that both codes are currently specified in RFC 9110 (HTTP Semantics), and that 308 was originally defined in RFC 7538, which RFC 9110 obsoletes." in 2 of 2 runs
Orchestrator-workers fell short of "States that both codes are currently specified in RFC 9110 (HTTP Semantics), and that 308 was originally defined in RFC 7538, which RFC 9110 obsoletes." in 2 of 2 runs
Plan-execute-verify fell short of "States that both codes are currently specified in RFC 9110 (HTTP Semantics), and that 308 was originally defined in RFC 7538, which RFC 9110 obsoletes." in 2 of 2 runs
Propose-select fell short of "States that both codes are currently specified in RFC 9110 (HTTP Semantics), and that 308 was originally defined in RFC 7538, which RFC 9110 obsoletes." in 2 of 2 runs
Gated-execute fell short of "States that both codes are currently specified in RFC 9110 (HTTP Semantics), and that 308 was originally defined in RFC 7538, which RFC 9110 obsoletes." in 1 of 2 runs

How the failures failed12wrong answer

In numbers, on Claude Haiku 4.5

  • Gated-execute
    0.917

    2 runs0% pass$0.064 / run32.4s

    wrong answer ×2

  • Orchestrator-workers
    0.917

    2 runs0% pass$0.207 / run1m 15s

    wrong answer ×2

  • Plan-execute
    0.885

    2 runs0% pass$0.125 / run1m 34s

    wrong answer ×2

  • Plan-execute-verify
    0.885

    2 runs0% pass$0.221 / run2m 35s

    wrong answer ×2

  • Single
    0.885

    2 runs0% pass$0.073 / run41.6s

    wrong answer ×2

  • Propose-select
    0.573

    2 runs0% pass$0.243 / run1m 52s

    wrong answer ×2

Every run on Haiku 4.512 runs · $1.86 spent
RunStrategyRepeatScoreResultCostLatency
claude-haiku-4-5-20251001__gated-execute__research-http-redirects__r1Gated-execute10.917wrong answer$0.06535.1s
claude-haiku-4-5-20251001__gated-execute__research-http-redirects__r2Gated-execute20.917wrong answer$0.06329.6s
claude-haiku-4-5-20251001__orchestrator-workers__research-http-redirects__r1Orchestrator-workers10.917wrong answer$0.1921m 14s
claude-haiku-4-5-20251001__orchestrator-workers__research-http-redirects__r2Orchestrator-workers20.917wrong answer$0.2211m 15s
claude-haiku-4-5-20251001__plan-execute-verify__research-http-redirects__r1Plan-execute-verify10.917wrong answer$0.2813m 22s
claude-haiku-4-5-20251001__plan-execute-verify__research-http-redirects__r2Plan-execute-verify20.854wrong answer$0.1601m 49s
claude-haiku-4-5-20251001__plan-execute__research-http-redirects__r1Plan-execute10.854wrong answer$0.1051m 9s
claude-haiku-4-5-20251001__plan-execute__research-http-redirects__r2Plan-execute20.917wrong answer$0.1461m 59s
claude-haiku-4-5-20251001__propose-select__research-http-redirects__r1Propose-select10.229wrong answer$0.2382m 21s
claude-haiku-4-5-20251001__propose-select__research-http-redirects__r2Propose-select20.917wrong answer$0.2471m 22s
claude-haiku-4-5-20251001__single__research-http-redirects__r1Single10.917wrong answer$0.09654.7s
claude-haiku-4-5-20251001__single__research-http-redirects__r2Single20.854wrong answer$0.05028.4s

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.