ScaffoldBench

research · web

async vs defer on <script>

mediumhtmlbrowsersynthesis
Task id
research-script-loading
Turn limit
30
Runs on Sonnet 5.5
12
Pass rate
83%

Pre-registered hypothesis

Hypothesis

Mostly memorable material, so expect high rubric agreement across architectures; the spread should come from grounding — whether each claim is backed by a verified quote — and from the module-script details, which only the HTML Standard states precisely.

Written when the task was written, before any run. It is a prediction in words, so read it against what happened rather than trusting a verdict drawn for you.

What happened on Sonnet 5.5

  • Highest mean score: Single 1.000
  • Lowest: Gated-execute 0.950

Means over the runs that count; intervals and verdicts are on the results page.

Who solved it

On Sonnet 5.5, Single, Plan-execute, Orchestrator-workers and Plan-execute-verify score highest at 1.000 and Gated-execute lowest at 0.950. The same task, every architecture, every model: one square per run.

Haiku 4.5
Sonnet 5.5
Opus 5.5
Single
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
Plan-execute
1.000 mean score of the runs that count
1.000 mean score of the runs that count
0.958 mean score of the runs that count
Orchestrator-workers
0.953 mean score of the runs that count
1.000 mean score of the runs that count
0.911 mean score of the runs that count
Plan-execute-verify
0.953 mean score of the runs that count
1.000 mean score of the runs that count
0.950 mean score of the runs that count
Propose-select
1.000 mean score of the runs that count
0.969 mean score of the runs that count
1.000 mean score of the runs that count
Gated-execute
0.953 mean score of the runs that count
0.950 mean score of the runs that count
1.000 mean score of the runs that count

Point at a square or tab to it to read its run. Each square opens its run.

passedpartial creditfailedexcluded: integrity check failedexcluded: API errorOne square per run; each opens its run.

The prompt every architecture was handed

In HTML, what exactly do the `async` and `defer` attributes on a <script> element do? Cover how each interacts with HTML parsing, when each executes relative to the DOMContentLoaded event, what each guarantees about execution order when there are several scripts, and how inline scripts and type="module" scripts differ from the classic src case. Research this with the provided web tools and write a complete, self-contained answer. Write the answer for the user in prose. Then list the evidence for your key claims: end the answer with one fenced JSON block of exactly this shape, one entry per claim: ```json {"evidence": [{"claim": "<the claim, in your words>", "quote": "<a sentence copied word for word from the page>", "url": "<the URL of that page, as you opened it>"}]} ``` Every quote is checked against the text of the page at its URL. A paraphrase, or a quote from a page you did not open with fetch_page, does not count as evidence. The block must be valid JSON: write a double quote inside a string as \".

The domain policy

Appended to the system prompt for every architecture, so none of them gets a rules advantage.

Research rules: - The `search` and `fetch_page` tools are your only access to the web. Any other browsing tool is unavailable. - Search snippets are previews, not sources. Open a page with `fetch_page` before relying on what it says. - Long pages arrive one window at a time. Continue with `offset`, or use `find` to jump to a phrase. - Cite the URL of every source you rely on, next to the claim it supports. - Never cite a URL you did not open with `fetch_page`. A citation to an unopened page is scored as a fabrication. - Quotes must be copied word for word from the page you cite; they are checked against the page text. - Quote only passages the tools have shown you in this session. A quote recalled from memory does not count, even when the page contains it. - When sources disagree, say so explicitly and attribute each position, instead of silently picking one.

The traps it sets

A trap is a specific way to go wrong, declared before any run and tied to the assertions or rules that register it. A trap fired in a run when one of those failed. On Sonnet 5.5:

async vs defer on <script>2 traps · runs that fell in, of runs ranked
Trap
Single
Plan-execute
Orchestrator-workers
Plan-execute-verify
Propose-select
Gated-execute
async-after-dclMDN's DOMContentLoaded page says the event fires 'before the execution of scripts with async'; the HTML Standard says an async script runs as soon as it is available, possibly before parsing completes.
Single fell into async-after-dcl in 0 of 2 runs
Plan-execute fell into async-after-dcl in 0 of 2 runs
Orchestrator-workers fell into async-after-dcl in 0 of 2 runs
Plan-execute-verify fell into async-after-dcl in 0 of 2 runs
Propose-select fell into async-after-dcl in 0 of 2 runs
Gated-execute fell into async-after-dcl in 0 of 2 runs
module-deferClaiming defer changes a module script; the HTML Standard says it has no effect on module scripts.
Single fell into module-defer in 0 of 2 runs
Plan-execute fell into module-defer in 0 of 2 runs
Orchestrator-workers fell into module-defer in 0 of 2 runs
Plan-execute-verify fell into module-defer in 0 of 2 runs
Propose-select fell into module-defer in 0 of 2 runs
Gated-execute fell into module-defer in 0 of 2 runs

Where it was lost

On Sonnet 5.5, the item most often missed is “States that async and defer have no effect on a classic inline script (a script element without a src attribute).”, by 8% of runs. A score says an architecture lost points. This says which ones: for each graded item, the runs that earned less than full credit, out of the runs that count.

Graded items on Sonnet 5.5runs that fell short, of runs that count
Graded item
Single
Plan-execute
Orchestrator-workers
Plan-execute-verify
Propose-select
Gated-execute
States that both async and defer let a classic script be fetched in parallel with HTML parsing, so the fetch does not block the parser, unlike a classic script with src and neither attribute.weight 1
Single fell short of "States that both async and defer let a classic script be fetched in parallel with HTML parsing, so the fetch does not block the parser, unlike a classic script with src and neither attribute." in 0 of 2 runs
Plan-execute fell short of "States that both async and defer let a classic script be fetched in parallel with HTML parsing, so the fetch does not block the parser, unlike a classic script with src and neither attribute." in 0 of 2 runs
Orchestrator-workers fell short of "States that both async and defer let a classic script be fetched in parallel with HTML parsing, so the fetch does not block the parser, unlike a classic script with src and neither attribute." in 0 of 2 runs
Plan-execute-verify fell short of "States that both async and defer let a classic script be fetched in parallel with HTML parsing, so the fetch does not block the parser, unlike a classic script with src and neither attribute." in 0 of 2 runs
Propose-select fell short of "States that both async and defer let a classic script be fetched in parallel with HTML parsing, so the fetch does not block the parser, unlike a classic script with src and neither attribute." in 0 of 2 runs
Gated-execute fell short of "States that both async and defer let a classic script be fetched in parallel with HTML parsing, so the fetch does not block the parser, unlike a classic script with src and neither attribute." in 0 of 2 runs
States that a defer script executes only after the document has finished parsing, and before DOMContentLoaded fires.weight 1.5
Single fell short of "States that a defer script executes only after the document has finished parsing, and before DOMContentLoaded fires." in 0 of 2 runs
Plan-execute fell short of "States that a defer script executes only after the document has finished parsing, and before DOMContentLoaded fires." in 0 of 2 runs
Orchestrator-workers fell short of "States that a defer script executes only after the document has finished parsing, and before DOMContentLoaded fires." in 0 of 2 runs
Plan-execute-verify fell short of "States that a defer script executes only after the document has finished parsing, and before DOMContentLoaded fires." in 0 of 2 runs
Propose-select fell short of "States that a defer script executes only after the document has finished parsing, and before DOMContentLoaded fires." in 0 of 2 runs
Gated-execute fell short of "States that a defer script executes only after the document has finished parsing, and before DOMContentLoaded fires." in 0 of 2 runs
States that an async script executes as soon as it is available, interrupting parsing if parsing is still in progress, so it may run before or after DOMContentLoaded.weight 1.5
Single fell short of "States that an async script executes as soon as it is available, interrupting parsing if parsing is still in progress, so it may run before or after DOMContentLoaded." in 0 of 2 runs
Plan-execute fell short of "States that an async script executes as soon as it is available, interrupting parsing if parsing is still in progress, so it may run before or after DOMContentLoaded." in 0 of 2 runs
Orchestrator-workers fell short of "States that an async script executes as soon as it is available, interrupting parsing if parsing is still in progress, so it may run before or after DOMContentLoaded." in 0 of 2 runs
Plan-execute-verify fell short of "States that an async script executes as soon as it is available, interrupting parsing if parsing is still in progress, so it may run before or after DOMContentLoaded." in 0 of 2 runs
Propose-select fell short of "States that an async script executes as soon as it is available, interrupting parsing if parsing is still in progress, so it may run before or after DOMContentLoaded." in 0 of 2 runs
Gated-execute fell short of "States that an async script executes as soon as it is available, interrupting parsing if parsing is still in progress, so it may run before or after DOMContentLoaded." in 0 of 2 runs
States that multiple defer scripts execute in document order, while async scripts execute in the order their downloads complete, which is not predictable.weight 1.5
Single fell short of "States that multiple defer scripts execute in document order, while async scripts execute in the order their downloads complete, which is not predictable." in 0 of 2 runs
Plan-execute fell short of "States that multiple defer scripts execute in document order, while async scripts execute in the order their downloads complete, which is not predictable." in 0 of 2 runs
Orchestrator-workers fell short of "States that multiple defer scripts execute in document order, while async scripts execute in the order their downloads complete, which is not predictable." in 0 of 2 runs
Plan-execute-verify fell short of "States that multiple defer scripts execute in document order, while async scripts execute in the order their downloads complete, which is not predictable." in 0 of 2 runs
Propose-select fell short of "States that multiple defer scripts execute in document order, while async scripts execute in the order their downloads complete, which is not predictable." in 0 of 2 runs
Gated-execute fell short of "States that multiple defer scripts execute in document order, while async scripts execute in the order their downloads complete, which is not predictable." in 0 of 2 runs
States that async and defer have no effect on a classic inline script (a script element without a src attribute).weight 1
Single fell short of "States that async and defer have no effect on a classic inline script (a script element without a src attribute)." in 0 of 2 runs
Plan-execute fell short of "States that async and defer have no effect on a classic inline script (a script element without a src attribute)." in 0 of 2 runs
Orchestrator-workers fell short of "States that async and defer have no effect on a classic inline script (a script element without a src attribute)." in 0 of 2 runs
Plan-execute-verify fell short of "States that async and defer have no effect on a classic inline script (a script element without a src attribute)." in 0 of 2 runs
Propose-select fell short of "States that async and defer have no effect on a classic inline script (a script element without a src attribute)." in 1 of 2 runs
Gated-execute fell short of "States that async and defer have no effect on a classic inline script (a script element without a src attribute)." in 0 of 2 runs
States that module scripts are deferred by default, so the defer attribute has no effect on them, while the async attribute does apply to module scripts.weight 1.5
Single fell short of "States that module scripts are deferred by default, so the defer attribute has no effect on them, while the async attribute does apply to module scripts." in 0 of 2 runs
Plan-execute fell short of "States that module scripts are deferred by default, so the defer attribute has no effect on them, while the async attribute does apply to module scripts." in 0 of 2 runs
Orchestrator-workers fell short of "States that module scripts are deferred by default, so the defer attribute has no effect on them, while the async attribute does apply to module scripts." in 0 of 2 runs
Plan-execute-verify fell short of "States that module scripts are deferred by default, so the defer attribute has no effect on them, while the async attribute does apply to module scripts." in 0 of 2 runs
Propose-select fell short of "States that module scripts are deferred by default, so the defer attribute has no effect on them, while the async attribute does apply to module scripts." in 0 of 2 runs
Gated-execute fell short of "States that module scripts are deferred by default, so the defer attribute has no effect on them, while the async attribute does apply to module scripts." in 0 of 2 runs
Cited pages that were never openedA rule: breaching it multiplies the score down.
Single fell short of "Cited pages that were never opened" in 0 of 2 runs
Plan-execute fell short of "Cited pages that were never opened" in 0 of 2 runs
Orchestrator-workers fell short of "Cited pages that were never opened" in 0 of 2 runs
Plan-execute-verify fell short of "Cited pages that were never opened" in 0 of 2 runs
Propose-select fell short of "Cited pages that were never opened" in 0 of 2 runs
Gated-execute fell short of "Cited pages that were never opened" in 1 of 2 runs

How the failures failed1hallucinated citation1wrong answer

In numbers, on Claude Sonnet 5.5

  • Orchestrator-workers
    1.000

    2 runs100% pass$0.620 / run1m 25s

  • Plan-execute
    1.000

    2 runs100% pass$0.196 / run58.8s

  • Plan-execute-verify
    1.000

    2 runs100% pass$0.328 / run1m 22s

  • Single
    1.000

    2 runs100% pass$0.137 / run37.6s

  • Propose-select
    0.969

    2 runs50% pass$0.587 / run1m 27s

    wrong answer ×1

  • Gated-execute
    0.950

    2 runs50% pass$0.132 / run36.1s

    hallucinated citation ×1

Every run on Sonnet 5.512 runs · $4.00 spent
RunStrategyRepeatScoreResultCostLatency
claude-sonnet-5-5__gated-execute__research-script-loading__r1Gated-execute11.000pass$0.12936.0s
claude-sonnet-5-5__gated-execute__research-script-loading__r2Gated-execute20.900hallucinated citation$0.13536.2s
claude-sonnet-5-5__orchestrator-workers__research-script-loading__r1Orchestrator-workers11.000pass$0.5911m 29s
claude-sonnet-5-5__orchestrator-workers__research-script-loading__r2Orchestrator-workers21.000pass$0.6491m 22s
claude-sonnet-5-5__plan-execute-verify__research-script-loading__r1Plan-execute-verify11.000pass$0.3501m 25s
claude-sonnet-5-5__plan-execute-verify__research-script-loading__r2Plan-execute-verify21.000pass$0.3061m 19s
claude-sonnet-5-5__plan-execute__research-script-loading__r1Plan-execute11.000pass$0.22356.1s
claude-sonnet-5-5__plan-execute__research-script-loading__r2Plan-execute21.000pass$0.1691m 2s
claude-sonnet-5-5__propose-select__research-script-loading__r1Propose-select11.000pass$0.6051m 40s
claude-sonnet-5-5__propose-select__research-script-loading__r2Propose-select20.938wrong answer$0.5701m 14s
claude-sonnet-5-5__single__research-script-loading__r1Single11.000pass$0.13736.9s
claude-sonnet-5-5__single__research-script-loading__r2Single21.000pass$0.13638.3s

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.