ScaffoldBench

research · web

Where ten HTTP header fields are defined

hardhttpspecificationsbreadthversionsgold-key
Task id
research-header-fields
Turn limit
40
Runs on Opus 5.5
12
Pass rate
75%

Pre-registered hypothesis

Hypothesis

The strongest case for fan-out, ten lookups across four current RFCs, but expect a tie on score: in a one-run calibration, single scored 1.0 with both Haiku 4.5 and Sonnet 5.5 once it read the corpus (closed-book, Haiku named an obsolete RFC for 8 of 10 fields). Fan-out should cut wall-clock time at a higher token cost.

Written when the task was written, before any run. It is a prediction in words, so read it against what happened rather than trusting a verdict drawn for you.

What happened on Opus 5.5

  • Highest mean score: Single 1.000
  • Lowest: Plan-execute-verify 0.867

Means over the runs that count; intervals and verdicts are on the results page.

Who solved it

On Opus 5.5, Single, Orchestrator-workers, Propose-select and Gated-execute score highest at 1.000 and Plan-execute-verify lowest at 0.867. The same task, every architecture, every model: one square per run.

Haiku 4.5
Sonnet 5.5
Opus 5.5
Single
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
Plan-execute
0.619 mean score of the runs that count
1.000 mean score of the runs that count
0.950 mean score of the runs that count
Orchestrator-workers
0.950 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
Plan-execute-verify
1.000 mean score of the runs that count
1.000 mean score of the runs that count
0.867 mean score of the runs that count
Propose-select
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
Gated-execute
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count

Point at a square or tab to it to read its run. Each square opens its run.

passedpartial creditfailedexcluded: integrity check failedexcluded: API errorOne square per run; each opens its run.

The prompt every architecture was handed

For each HTTP header field below, find the RFC that currently defines it and the section of that RFC that contains its definition. An RFC that has since been obsoleted does not count. - Age - Cache-Control - Expires - Last-Modified - ETag - Location - Retry-After - Vary - Strict-Transport-Security - Link Research this with the provided web tools. Report, for every header field: - rfc: the number of the RFC that currently defines the field - section: the number of the section of that RFC that defines the field (just the number, e.g. "12.3.4") - quote: a sentence from the section of that RFC that defines the field, copied word for word (at least six words). - url: the URL of the page the quote comes from, which you opened with fetch_page. Write whatever explanation you like, then end the answer with one fenced JSON block of exactly this shape, with one entry per header field: ```json {"answers":[{"field":"Age","rfc":"…","section":"…","quote":"<a sentence copied word for word>","url":"<the URL of that page, as you opened it>"}]} ``` Quotes are checked against the text of the page at their URL, so copy them exactly. The block must be valid JSON: write a double quote inside a string as \".

The domain policy

Appended to the system prompt for every architecture, so none of them gets a rules advantage.

Research rules: - The `search` and `fetch_page` tools are your only access to the web. Any other browsing tool is unavailable. - Search snippets are previews, not sources. Open a page with `fetch_page` before relying on what it says. - Long pages arrive one window at a time. Continue with `offset`, or use `find` to jump to a phrase. - Cite the URL of every source you rely on, next to the claim it supports. - Never cite a URL you did not open with `fetch_page`. A citation to an unopened page is scored as a fabrication. - Quotes must be copied word for word from the page you cite; they are checked against the page text. - Quote only passages the tools have shown you in this session. A quote recalled from memory does not count, even when the page contains it. - When sources disagree, say so explicitly and attribute each position, instead of silently picking one.

The traps it sets

A trap is a specific way to go wrong, declared before any run and tied to the assertions or rules that register it. A trap fired in a run when one of those failed. On Opus 5.5:

Where ten HTTP header fields are defined2 traps · runs that fell in, of runs ranked
Trap
Single
Plan-execute
Orchestrator-workers
Plan-execute-verify
Propose-select
Gated-execute
obsolete-rfcRFC 7231, 7232 and 7234 define Location, Retry-After, Vary, ETag, Last-Modified, Age, Cache-Control and Expires, and are obsoleted by RFC 9110 and 9111.
Single fell into obsolete-rfc in 0 of 2 runs
Plan-execute fell into obsolete-rfc in 0 of 2 runs
Orchestrator-workers fell into obsolete-rfc in 0 of 2 runs
Plan-execute-verify fell into obsolete-rfc in 0 of 2 runs
Propose-select fell into obsolete-rfc in 0 of 2 runs
Gated-execute fell into obsolete-rfc in 0 of 2 runs
same-section-numbersRFC 7234 numbers Age, Cache-Control and Expires 5.1, 5.2 and 5.3, exactly like RFC 9111, so the section alone does not reveal an obsolete source; the quote must come from RFC 9111.
Single fell into same-section-numbers in 0 of 2 runs
Plan-execute fell into same-section-numbers in 0 of 2 runs
Orchestrator-workers fell into same-section-numbers in 0 of 2 runs
Plan-execute-verify fell into same-section-numbers in 0 of 2 runs
Propose-select fell into same-section-numbers in 0 of 2 runs
Gated-execute fell into same-section-numbers in 0 of 2 runs

Where it was lost

On Opus 5.5, the item most often missed is “Cited pages that were never opened”, by 25% of runs. A score says an architecture lost points. This says which ones: for each graded item, the runs that earned less than full credit, out of the runs that count.

Graded items on Opus 5.5runs that fell short, of runs that count
Graded item
Single
Plan-execute
Orchestrator-workers
Plan-execute-verify
Propose-select
Gated-execute
header field Ageweight 1
Single fell short of "header field Age" in 0 of 2 runs
Plan-execute fell short of "header field Age" in 0 of 2 runs
Orchestrator-workers fell short of "header field Age" in 0 of 2 runs
Plan-execute-verify fell short of "header field Age" in 0 of 2 runs
Propose-select fell short of "header field Age" in 0 of 2 runs
Gated-execute fell short of "header field Age" in 0 of 2 runs
header field Cache-Controlweight 1
Single fell short of "header field Cache-Control" in 0 of 2 runs
Plan-execute fell short of "header field Cache-Control" in 0 of 2 runs
Orchestrator-workers fell short of "header field Cache-Control" in 0 of 2 runs
Plan-execute-verify fell short of "header field Cache-Control" in 0 of 2 runs
Propose-select fell short of "header field Cache-Control" in 0 of 2 runs
Gated-execute fell short of "header field Cache-Control" in 0 of 2 runs
header field Expiresweight 1
Single fell short of "header field Expires" in 0 of 2 runs
Plan-execute fell short of "header field Expires" in 0 of 2 runs
Orchestrator-workers fell short of "header field Expires" in 0 of 2 runs
Plan-execute-verify fell short of "header field Expires" in 0 of 2 runs
Propose-select fell short of "header field Expires" in 0 of 2 runs
Gated-execute fell short of "header field Expires" in 0 of 2 runs
header field Last-Modifiedweight 1
Single fell short of "header field Last-Modified" in 0 of 2 runs
Plan-execute fell short of "header field Last-Modified" in 0 of 2 runs
Orchestrator-workers fell short of "header field Last-Modified" in 0 of 2 runs
Plan-execute-verify fell short of "header field Last-Modified" in 0 of 2 runs
Propose-select fell short of "header field Last-Modified" in 0 of 2 runs
Gated-execute fell short of "header field Last-Modified" in 0 of 2 runs
header field ETagweight 1
Single fell short of "header field ETag" in 0 of 2 runs
Plan-execute fell short of "header field ETag" in 0 of 2 runs
Orchestrator-workers fell short of "header field ETag" in 0 of 2 runs
Plan-execute-verify fell short of "header field ETag" in 0 of 2 runs
Propose-select fell short of "header field ETag" in 0 of 2 runs
Gated-execute fell short of "header field ETag" in 0 of 2 runs
header field Locationweight 1
Single fell short of "header field Location" in 0 of 2 runs
Plan-execute fell short of "header field Location" in 0 of 2 runs
Orchestrator-workers fell short of "header field Location" in 0 of 2 runs
Plan-execute-verify fell short of "header field Location" in 0 of 2 runs
Propose-select fell short of "header field Location" in 0 of 2 runs
Gated-execute fell short of "header field Location" in 0 of 2 runs
header field Retry-Afterweight 1
Single fell short of "header field Retry-After" in 0 of 2 runs
Plan-execute fell short of "header field Retry-After" in 0 of 2 runs
Orchestrator-workers fell short of "header field Retry-After" in 0 of 2 runs
Plan-execute-verify fell short of "header field Retry-After" in 0 of 2 runs
Propose-select fell short of "header field Retry-After" in 0 of 2 runs
Gated-execute fell short of "header field Retry-After" in 0 of 2 runs
header field Varyweight 1
Single fell short of "header field Vary" in 0 of 2 runs
Plan-execute fell short of "header field Vary" in 0 of 2 runs
Orchestrator-workers fell short of "header field Vary" in 0 of 2 runs
Plan-execute-verify fell short of "header field Vary" in 0 of 2 runs
Propose-select fell short of "header field Vary" in 0 of 2 runs
Gated-execute fell short of "header field Vary" in 0 of 2 runs
header field Strict-Transport-Securityweight 1
Single fell short of "header field Strict-Transport-Security" in 0 of 2 runs
Plan-execute fell short of "header field Strict-Transport-Security" in 0 of 2 runs
Orchestrator-workers fell short of "header field Strict-Transport-Security" in 0 of 2 runs
Plan-execute-verify fell short of "header field Strict-Transport-Security" in 0 of 2 runs
Propose-select fell short of "header field Strict-Transport-Security" in 0 of 2 runs
Gated-execute fell short of "header field Strict-Transport-Security" in 0 of 2 runs
header field Linkweight 1
Single fell short of "header field Link" in 0 of 2 runs
Plan-execute fell short of "header field Link" in 0 of 2 runs
Orchestrator-workers fell short of "header field Link" in 0 of 2 runs
Plan-execute-verify fell short of "header field Link" in 0 of 2 runs
Propose-select fell short of "header field Link" in 0 of 2 runs
Gated-execute fell short of "header field Link" in 0 of 2 runs
Cited pages that were never openedA rule: breaching it multiplies the score down.
Single fell short of "Cited pages that were never opened" in 0 of 2 runs
Plan-execute fell short of "Cited pages that were never opened" in 1 of 2 runs
Orchestrator-workers fell short of "Cited pages that were never opened" in 0 of 2 runs
Plan-execute-verify fell short of "Cited pages that were never opened" in 2 of 2 runs
Propose-select fell short of "Cited pages that were never opened" in 0 of 2 runs
Gated-execute fell short of "Cited pages that were never opened" in 0 of 2 runs

How the failures failed3hallucinated citation

In numbers, on Claude Opus 5.5

  • Gated-execute
    1.000

    2 runs100% pass$0.328 / run45.5s

  • Orchestrator-workers
    1.000

    2 runs100% pass$0.755 / run1m 2s

  • Propose-select
    1.000

    2 runs100% pass$1.44 / run1m 35s

  • Single
    1.000

    2 runs100% pass$0.330 / run43.9s

  • Plan-execute
    0.950

    2 runs50% pass$0.508 / run59.7s

    hallucinated citation ×1

  • Plan-execute-verify
    0.867

    2 runs0% pass$0.698 / run1m 37s

    hallucinated citation ×2

Every run on Opus 5.512 runs · $8.12 spent
RunStrategyRepeatScoreResultCostLatency
claude-opus-5-5__gated-execute__research-header-fields__r1Gated-execute11.000pass$0.31038.7s
claude-opus-5-5__gated-execute__research-header-fields__r2Gated-execute21.000pass$0.34552.4s
claude-opus-5-5__orchestrator-workers__research-header-fields__r1Orchestrator-workers11.000pass$0.7631m 2s
claude-opus-5-5__orchestrator-workers__research-header-fields__r2Orchestrator-workers21.000pass$0.7471m 2s
claude-opus-5-5__plan-execute-verify__research-header-fields__r1Plan-execute-verify10.833hallucinated citation$0.7161m 41s
claude-opus-5-5__plan-execute-verify__research-header-fields__r2Plan-execute-verify20.900hallucinated citation$0.6811m 33s
claude-opus-5-5__plan-execute__research-header-fields__r1Plan-execute10.900hallucinated citation$0.46654.8s
claude-opus-5-5__plan-execute__research-header-fields__r2Plan-execute21.000pass$0.5501m 5s
claude-opus-5-5__propose-select__research-header-fields__r1Propose-select11.000pass$1.501m 36s
claude-opus-5-5__propose-select__research-header-fields__r2Propose-select21.000pass$1.381m 34s
claude-opus-5-5__single__research-header-fields__r1Single11.000pass$0.30548.7s
claude-opus-5-5__single__research-header-fields__r2Single21.000pass$0.35639.2s

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.