ScaffoldBench

RunOpus 5.5repeat 1

Plan-execute-verify on async vs defer on <script>

claude-opus-5-5__plan-execute-verify__research-script-loading__r1

Score

0.900

hallucinated citation
design ran as designed
Cost
$0.702
+ $0.090 judging
Latency
2m 20s
setup 0.0s, grading 4.9s
Turns
41
Tool calls
36
Subagents
0
Tokens in / out
292k / 13k
235,887 read from cache

Did the design run?

every check held
  • Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
  • No API failures: heldharness.apiNo model call failed at the API.
  • Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
  • No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
  • Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
  • No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
  • Builtin tools: heldharness.builtin-toolsA check recorded with the run.
  • Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
  • No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
  • Judge answered: heldharness.judgeAt least one grading call returned a usable verdict.
  • The plan was a plan: heldplan-execute-verify.planThe planner’s reply was read through its schema and held at least one step.
  • Every verdict was read: heldplan-execute-verify.verdictsEvery review round ended with a verdict read from the schema reply, not a default.
  • The loop did what its verdicts said: heldplan-execute-verify.loopEvery verdict that called for a revision got one, and nothing was revised without such a verdict.
  • Each call was shown the right tools: heldplan-execute-verify.toolsThe planner was shown no tools, the verifier the read-only ones, and the executor and reviser the full toolset.

Phases

Plan: 0.0s to 16.7s, finished, 0 tool calls, $0.035
Execute: 16.7s to 1m 38s, finished, 21 tool calls, $0.437
Verify: 1m 38s to 2m 20s, finished, 15 tool calls, $0.230

Drawn from the phase records: timing and cost per call, no tool calls or messages.

PhaseDurationCostTurnsToolsModel
Plan16.7s$0.03520claude-opus-5-5
Execute1m 21s$0.4372221claude-opus-5-5
Verify42.3s$0.2301715claude-opus-5-5

The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.

Runs you can replay

Record this cell yourself: npm run bench -- run --strategies plan-execute-verify --tasks research-script-loading --models claude-opus-5-5 --repeats 1

Score breakdown

  • passStates that both async and defer let a classic script be fetched in parallel with HTML parsing, so the fetch does not block the parser, unlike a classic script with src and neither attribute.
    1.00weight 0.13

    met, backed by verified evidence E1, E2, E3. “the script downloads while the parser keeps going. The spec says it "will be fetched in parallel and evaluated when the page has finished parsing"”

  • passStates that a defer script executes only after the document has finished parsing, and before DOMContentLoaded fires.
    1.00weight 0.19

    met, backed by verified evidence E3, E4. “it runs after parsing finishes and before DOMContentLoaded.”

  • passStates that an async script executes as soon as it is available, interrupting parsing if parsing is still in progress, so it may run before or after DOMContentLoaded.
    1.00weight 0.19

    met, backed by verified evidence E2. “The spec's "potentially before parsing completes" means an async script can run before or after the event.”

  • passStates that multiple defer scripts execute in document order, while async scripts execute in the order their downloads complete, which is not predictable.
    1.00weight 0.19

    met, backed by verified evidence E5, E6, E7, E8. “**Order:** none is guaranteed. The spec puts async scripts in a *set* of scripts that will execute as soon as possible”

  • passStates that async and defer have no effect on a classic inline script (a script element without a src attribute).
    1.00weight 0.13

    met, backed by verified evidence E14, E15. “On a classic script without `src`, `async` and `defer` are not allowed and do nothing”

  • passStates that module scripts are deferred by default, so the defer attribute has no effect on them, while the async attribute does apply to module scripts.
    1.00weight 0.19

    met, backed by verified evidence E17, E18, E19, E20. “**`defer`:** has no effect on modules, according to the spec and MDN.”

Penalties

  • hallucinated citationCited pages that were never opened×0.9

    1 of 5 cited URLs were never opened: https://html.spec.whatwg.org/multipage/parsing.html

Scorer metadata

corpus
{"version":2,"snapshot":"2026-10-01","hash":"a5d703ba820ee9c957565ac15b9f98daeac8212372d73d7658a7a6bb2bbc1978","documents":6}
opened documents
["whatwg-html-scripting","mdn-script-element","webdev-optimize-resource-loading","mdn-domcontentloaded"]
opened by role
{"primary":["whatwg-html-scripting"],"secondary":["mdn-script-element","webdev-optimize-resource-loading","mdn-domcontentloaded"]}
displayed share
{"mdn-script-element":0.8566,"whatwg-html-scripting":0.2681,"webdev-third-party-javascript":0.0284,"webdev-optimize-resource-loading":0.3583,"mdn-domcontentloaded":1}
cited count
5
verified count
4
hallucinated count
1
citation integrity
0.8 (80%)
hallucinated urls
["https://html.spec.whatwg.org/multipage/parsing.html"]
evidence block
true
evidence block repaired
false
evidence count
20
verified quote count
20
verified quote rate
1
evidence
[{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"A classic external script with neither attribute blocks parsing while it is fetched and run.","quote":"If neither attribute is present, then the script is fetched and evaluated immediately, blocking parsing until these are both complete.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"An async classic script is fetched in parallel with parsing and runs as soon as it is available, possibly before parsing finishes.","quote":"For external classic scripts, if the async attribute is present, then the classic script will be fetched in parallel to parsing and evaluated as soon as it is available (potentially before parsing completes).","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"A defer classic script is fetched in parallel and runs when parsing has finished.","quote":"If the async attribute is not present but the defer attribute is present, then the classic script will be fetched in parallel and evaluated when the page has finished parsing.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"Deferred scripts run after parsing and before DOMContentLoaded, and they hold the event back.","quote":"Scripts with the defer attribute will prevent the DOMContentLoaded event from firing until the script has loaded and finished evaluating.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"Deferred scripts run in document order.","quote":"Scripts with the defer attribute will execute in the order in which they appear in the document.","verified":true,"document":"mdn-script-element"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"In the spec, deferred scripts and non-async modules go into one ordered per-document list that runs after parsing.","quote":"Append el to its parser document's list of scripts that will execute when the document has finished parsing.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Async scripts go into an unordered set and each runs when it is ready.","quote":"Let scripts be el's preparation-time document's set of scripts that will execute as soon as possible.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"Async scripts have no guaranteed order and block the page while they run.","quote":"scripts with an async attribute will execute as soon as the download is complete. This blocks the page and does not guarantee any specific execution order.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event","claim":"DOMContentLoaded waits for deferred and module scripts but not for async scripts.","quote":"It doesn't wait for other things like images, subframes, and async scripts to finish loading.","verified":true,"document":"mdn-domcontentloaded"},{"url":"https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event","claim":"MDN's DOMContentLoaded page says the event always fires before async scripts run, which is stronger than the spec.","quote":"DOMContentLoaded is fired after the execution of these scripts, but before the execution of scripts with async.","verified":true,"document":"mdn-domcontentloaded"},{"url":"https://web.dev/learn/performance/optimize-resource-loading","claim":"web.dev says defer scripts run at the same time as DOMContentLoaded, which is less precise than the spec and MDN.","quote":"Scripts loaded with async are parsed and executed immediately once downloaded, while scripts loaded with defer are executed when HTML document parsing is finished—this occurs at the same time as the browser's DOMContentLoaded event.","verified":true,"document":"webdev-optimize-resource-loading"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"If both attributes are present, async wins.","quote":"If the attribute is specified with the async attribute, the element will act as if only the async attribute is specified.","verified":true,"document":"mdn-script-element"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"defer next to async is a fallback for old browsers that only support defer.","quote":"The defer attribute may be specified even if the async attribute is specified, to cause legacy web browsers that only support defer (and not async) to fall back to the defer behavior instead of the blocking behavior that is the default.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"async/defer must not be used on inline classic scripts and do nothing there.","quote":"Warning: This attribute must not be used if the src attribute is absent (i.e., for inline scripts), in this case it would have no effect.","verified":true,"document":"mdn-script-element"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"The async/defer branches only apply to classic scripts with src, or to modules (including inline modules).","quote":"If el's type is \"classic\" and el has a src attribute, or el's type is \"module\":","verified":true,"document":"whatwg-html-scripting"},{"url":"https://web.dev/learn/performance/optimize-resource-loading","claim":"Inline classic scripts block the parser.","quote":"Similarly, inline scripts block the parser until the script is parsed and executed.","verified":true,"document":"webdev-optimize-resource-loading"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Without async, module scripts and their dependencies are fetched in parallel and run after parsing, which is deferred behavior.","quote":"Otherwise, the module script and its dependencies will be fetched in parallel to parsing and evaluated when the page has finished parsing.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"defer has no effect on modules because they are deferred by default.","quote":"The defer attribute has no effect on module scripts — they defer by default.","verified":true,"document":"mdn-script-element"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"An async module is fetched with its dependencies in parallel and runs as soon as it is available.","quote":"For module scripts, if the async attribute is present, then the module script and all its dependencies will be fetched in parallel to parsing, and the module script will be evaluated as soon as it is available (potentially before parsing completes).","verified":true,"document":"whatwg-html-scripting"},{"url":"https://web.dev/learn/performance/optimize-resource-loading","claim":"Module scripts, including inline ones, are deferred automatically.","quote":"async and defer allow external scripts to load without blocking the HTML parser while scripts (including inline scripts) with type=\"module\" are deferred automatically.","verified":true,"document":"webdev-optimize-resource-loading"}]
judge unverified quotes
0
judge verdicts
[{"parallel-download":{"status":"met","support":["E1","E2","E3"]},"defer-timing":{"status":"met","support":["E3","E4"]},"async-timing":{"status":"met","support":["E2"]},"ordering":{"status":"met","support":["E5","E6","E7","E8"]},"inline-ignored":{"status":"met","support":["E14","E15"]},"modules":{"status":"met","support":["E17","E18","E19","E20"]}},{"parallel-download":{"status":"met","support":["E1","E2","E3"]},"defer-timing":{"status":"met","support":["E3","E4"]},"async-timing":{"status":"met","support":["E2","E8"]},"ordering":{"status":"met","support":["E5","E6","E7","E8"]},"inline-ignored":{"status":"met","support":["E14","E15"]},"modules":{"status":"met","support":["E17","E18","E19","E20"]}},{"parallel-download":{"status":"met","support":["E1","E2","E3"]},"defer-timing":{"status":"met","support":["E3","E4"]},"async-timing":{"status":"met","support":["E2"]},"ordering":{"status":"met","support":["E5","E6","E7","E8"]},"inline-ignored":{"status":"met","support":["E14","E15"]},"modules":{"status":"met","support":["E17","E18","E19","E20"]}}]
judge failures
0
grounded weight share
1

Final answer

I have enough material and am writing up the answer now.

What async and defer do on <script>

Default: a classic external script with no attributes

A plain <script src="…"> stops the parser. The WHATWG spec says: "If neither attribute is present, then the script is fetched and evaluated immediately, blocking parsing until these are both complete." (https://html.spec.whatwg.org/multipage/scripting.html)

defer on a classic external script

async on a classic external script

Where the sources disagree:

  • MDN's DOMContentLoaded page says the event fires "before the execution of scripts with async." It also says an async-included script will find that DOMContentLoaded "has already fired." Both statements are stronger than the spec, which allows an async script to run before parsing finishes. The cautious reading is that the order is not guaranteed either way.
  • web.dev says deferred scripts run "at the same time as" DOMContentLoaded (https://web.dev/learn/performance/optimize-resource-loading). The spec and MDN are more precise: deferred scripts run first, then the event fires.
  • MDN's own summary says deferred scripts run "once everything has finished loading." That is looser than its attribute reference, which says after parsing and before DOMContentLoaded.

Both async and defer set

async wins. MDN says the element acts "as if only the async attribute is specified." The spec says you may still add defer so that old browsers that only support defer fall back to it instead of blocking (https://html.spec.whatwg.org/multipage/scripting.html).

Inline classic scripts

On a classic script without src, async and defer are not allowed and do nothing (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script). The spec's attribute table marks both as not allowed for inline classic scripts. Its algorithm only uses the async/defer branches when the script is classic with a src, or is a module. An inline classic script normally runs immediately and blocks the parser. One exception: if a parser-inserted script is waiting on a style sheet that blocks scripts, it becomes the pending parsing-blocking script and runs once that style sheet loads (https://html.spec.whatwg.org/multipage/scripting.html). web.dev: "inline scripts block the parser until the script is parsed and executed."

type="module"

  • Default: deferred, whether external or inline. The module and its dependencies are fetched while parsing continues and run when parsing has finished. In the spec, modules go into the same deferred list as defer scripts, so they run in document order alongside them and before DOMContentLoaded. MDN's DOMContentLoaded page lists <script type="module"> among the deferred scripts the event waits for.
  • defer: has no effect on modules, according to the spec and MDN.
  • async: the module and all its dependencies are fetched in parallel, and the module runs as soon as it is available, possibly before parsing completes. Unlike inline classic scripts, inline modules do respect async. The spec's table allows async on inline modules, and web.dev notes that inline modules are also deferred automatically.

Two notes on sources

  • The WHATWG parsing page (https://html.spec.whatwg.org/multipage/parsing.html) returned 404. I couldn't quote the spec's "the end" steps, so the DOMContentLoaded timing above relies on MDN plus the scripting chapter.
  • Some spec sentences in the plan weren't on the page word for word, for example "For classic scripts, if the async attribute is present". The current text says "For external classic scripts…". I quoted the actual wording.
{"evidence": [
{"claim": "A classic external script with neither attribute blocks parsing while it is fetched and run.", "quote": "If neither attribute is present, then the script is fetched and evaluated immediately, blocking parsing until these are both complete.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "An async classic script is fetched in parallel with parsing and runs as soon as it is available, possibly before parsing finishes.", "quote": "For external classic scripts, if the async attribute is present, then the classic script will be fetched in parallel to parsing and evaluated as soon as it is available (potentially before parsing completes).", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "A defer classic script is fetched in parallel and runs when parsing has finished.", "quote": "If the async attribute is not present but the defer attribute is present, then the classic script will be fetched in parallel and evaluated when the page has finished parsing.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Deferred scripts run after parsing and before DOMContentLoaded, and they hold the event back.", "quote": "Scripts with the defer attribute will prevent the DOMContentLoaded event from firing until the script has loaded and finished evaluating.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "Deferred scripts run in document order.", "quote": "Scripts with the defer attribute will execute in the order in which they appear in the document.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "In the spec, deferred scripts and non-async modules go into one ordered per-document list that runs after parsing.", "quote": "Append el to its parser document's list of scripts that will execute when the document has finished parsing.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Async scripts go into an unordered set and each runs when it is ready.", "quote": "Let scripts be el's preparation-time document's set of scripts that will execute as soon as possible.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Async scripts have no guaranteed order and block the page while they run.", "quote": "scripts with an async attribute will execute as soon as the download is complete. This blocks the page and does not guarantee any specific execution order.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "DOMContentLoaded waits for deferred and module scripts but not for async scripts.", "quote": "It doesn't wait for other things like images, subframes, and async scripts to finish loading.", "url": "https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event"},
{"claim": "MDN's DOMContentLoaded page says the event always fires before async scripts run, which is stronger than the spec.", "quote": "DOMContentLoaded is fired after the execution of these scripts, but before the execution of scripts with async.", "url": "https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event"},
{"claim": "web.dev says defer scripts run at the same time as DOMContentLoaded, which is less precise than the spec and MDN.", "quote": "Scripts loaded with async are parsed and executed immediately once downloaded, while scripts loaded with defer are executed when HTML document parsing is finished—this occurs at the same time as the browser's DOMContentLoaded event.", "url": "https://web.dev/learn/performance/optimize-resource-loading"},
{"claim": "If both attributes are present, async wins.", "quote": "If the attribute is specified with the async attribute, the element will act as if only the async attribute is specified.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "defer next to async is a fallback for old browsers that only support defer.", "quote": "The defer attribute may be specified even if the async attribute is specified, to cause legacy web browsers that only support defer (and not async) to fall back to the defer behavior instead of the blocking behavior that is the default.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "async/defer must not be used on inline classic scripts and do nothing there.", "quote": "Warning: This attribute must not be used if the src attribute is absent (i.e., for inline scripts), in this case it would have no effect.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "The async/defer branches only apply to classic scripts with src, or to modules (including inline modules).", "quote": "If el's type is \"classic\" and el has a src attribute, or el's type is \"module\":", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Inline classic scripts block the parser.", "quote": "Similarly, inline scripts block the parser until the script is parsed and executed.", "url": "https://web.dev/learn/performance/optimize-resource-loading"},
{"claim": "Without async, module scripts and their dependencies are fetched in parallel and run after parsing, which is deferred behavior.", "quote": "Otherwise, the module script and its dependencies will be fetched in parallel to parsing and evaluated when the page has finished parsing.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "defer has no effect on modules because they are deferred by default.", "quote": "The defer attribute has no effect on module scripts — they defer by default.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "An async module is fetched with its dependencies in parallel and runs as soon as it is available.", "quote": "For module scripts, if the async attribute is present, then the module script and all its dependencies will be fetched in parallel to parsing, and the module script will be evaluated as soon as it is available (potentially before parsing completes).", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Module scripts, including inline ones, are deferred automatically.", "quote": "async and defer allow external scripts to load without blocking the HTML parser while scripts (including inline scripts) with type=\"module\" are deferred automatically.", "url": "https://web.dev/learn/performance/optimize-resource-loading"}
]}

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.