ScaffoldBench

RunOpus 5.5repeat 2

Plan-execute on async vs defer on <script>

claude-opus-5-5__plan-execute__research-script-loading__r2

Score

0.917

hallucinated citation
design ran as designed
Cost
$0.382
+ $0.085 judging
Latency
1m 13s
setup 0.0s, grading 5.7s
Turns
16
Tool calls
13
Subagents
0
Tokens in / out
142k / 7.6k
113,275 read from cache

Did the design run?

every check held
  • Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
  • No API failures: heldharness.apiNo model call failed at the API.
  • Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
  • No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
  • Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
  • No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
  • Builtin tools: heldharness.builtin-toolsA check recorded with the run.
  • Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
  • No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
  • Judge answered: heldharness.judgeAt least one grading call returned a usable verdict.
  • The plan was a plan: heldplan-execute.planThe planner’s reply was read through its schema and held at least one step.
  • Each call was shown the right tools: heldplan-execute.toolsThe planner was shown no tools and the executor the task’s full toolset.

Phases

Plan: 0.0s to 17.1s, finished, 0 tool calls, $0.042
Execute: 17.1s to 1m 13s, finished, 13 tool calls, $0.340

Drawn from the phase records: timing and cost per call, no tool calls or messages.

PhaseDurationCostTurnsToolsModel
Plan17.1s$0.04220claude-opus-5-5
Execute55.8s$0.3401413claude-opus-5-5

The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.

Runs you can replay

Record this cell yourself: npm run bench -- run --strategies plan-execute --tasks research-script-loading --models claude-opus-5-5 --repeats 1

Score breakdown

  • passStates that both async and defer let a classic script be fetched in parallel with HTML parsing, so the fetch does not block the parser, unlike a classic script with src and neither attribute.
    1.00weight 0.13

    met, backed by verified evidence E1, E2, E3. “both `async` and `defer` let the browser download the script while it keeps parsing the HTML”

  • passStates that a defer script executes only after the document has finished parsing, and before DOMContentLoaded fires.
    1.00weight 0.19

    met, backed by verified evidence E2, E4, E5. “A `defer` script waits until parsing has finished and runs before `DOMContentLoaded`”

  • passStates that an async script executes as soon as it is available, interrupting parsing if parsing is still in progress, so it may run before or after DOMContentLoaded.
    1.00weight 0.19

    met, backed by verified evidence E3, E7. “fetched in parallel to parsing and evaluated as soon as it is available (potentially before parsing completes)”

  • passStates that multiple defer scripts execute in document order, while async scripts execute in the order their downloads complete, which is not predictable.
    1.00weight 0.19

    met, backed by verified evidence E6, E7. “several `defer` scripts run in the order they appear in the document. An `async` script runs as soon as it has downloaded, in no guaranteed order”

  • passStates that async and defer have no effect on a classic inline script (a script element without a src attribute).
    1.00weight 0.13

    met, backed by verified evidence E11, E12. “Neither attribute does anything on an inline classic script.”

  • passStates that module scripts are deferred by default, so the defer attribute has no effect on them, while the async attribute does apply to module scripts.
    1.00weight 0.19

    met, backed by verified evidence E13, E14. “Module scripts are deferred by default, so `defer` does nothing on them, while `async` works on both external and inline modules.”

Penalties

  • hallucinated citationCited pages that were never opened×0.9166666666666667

    1 of 6 cited URLs were never opened: https://html.spec.whatwg.org/multipage/parsing.html

Scorer metadata

corpus
{"version":2,"snapshot":"2026-10-01","hash":"a5d703ba820ee9c957565ac15b9f98daeac8212372d73d7658a7a6bb2bbc1978","documents":6}
opened documents
["whatwg-html-scripting","mdn-script-element","mdn-domcontentloaded","webdev-third-party-javascript","webdev-optimize-resource-loading"]
opened by role
{"primary":["whatwg-html-scripting"],"secondary":["mdn-script-element","mdn-domcontentloaded","webdev-third-party-javascript","webdev-optimize-resource-loading"]}
displayed share
{"whatwg-html-scripting":0.2208,"webdev-optimize-resource-loading":0.3583,"mdn-script-element":0.549,"webdev-third-party-javascript":0.9407,"mdn-domcontentloaded":1}
cited count
6
verified count
5
hallucinated count
1
citation integrity
0.8333333333333334 (83%)
hallucinated urls
["https://html.spec.whatwg.org/multipage/parsing.html"]
evidence block
true
evidence block repaired
false
evidence count
15
verified quote count
15
verified quote rate
1
evidence
[{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"With neither async nor defer, a classic external script is fetched and run immediately, blocking the parser.","quote":"If neither attribute is present, then the script is fetched and evaluated immediately, blocking parsing until these are both complete.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"A deferred external classic script is fetched in parallel and runs after parsing finishes.","quote":"If the async attribute is not present but the defer attribute is present, then the classic script will be fetched in parallel and evaluated when the page has finished parsing.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"An async external classic script is fetched in parallel and runs as soon as it is available, possibly before parsing completes.","quote":"For external classic scripts, if the async attribute is present, then the classic script will be fetched in parallel to parsing and evaluated as soon as it is available (potentially before parsing completes).","verified":true,"document":"whatwg-html-scripting"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"Deferred scripts run after parsing but before DOMContentLoaded.","quote":"This Boolean attribute is set to indicate to a browser that the script is meant to be executed after the document has been parsed, but before firing DOMContentLoaded event.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"Deferred scripts hold back DOMContentLoaded until they finish.","quote":"Scripts with the defer attribute will prevent the DOMContentLoaded event from firing until the script has loaded and finished evaluating.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"Deferred scripts run in document order.","quote":"Scripts with the defer attribute will execute in the order in which they appear in the document.","verified":true,"document":"mdn-script-element"},{"url":"https://web.dev/articles/efficiently-load-third-party-javascript","claim":"Async scripts likely run out of document order and can interrupt parsing.","quote":"This means it's possible (and likely) that async scripts won't run in the order in which they appear in the HTML.","verified":true,"document":"webdev-third-party-javascript"},{"url":"https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event","claim":"DOMContentLoaded waits for parsing and deferred/module scripts but not for async scripts.","quote":"It doesn't wait for other things like images, subframes, and async scripts to finish loading.","verified":true,"document":"mdn-domcontentloaded"},{"url":"https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event","claim":"MDN's DOMContentLoaded note says async scripts run after the event, which conflicts with the spec's 'potentially before parsing completes'.","quote":"DOMContentLoaded is fired after the execution of these scripts, but before the execution of scripts with async.","verified":true,"document":"mdn-domcontentloaded"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"If both attributes are set, async takes precedence and defer is only a legacy fallback.","quote":"The defer attribute may be specified even if the async attribute is specified, to cause legacy web browsers that only support defer (and not async) to fall back to the defer behavior instead of the blocking behavior that is the default.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"async has no effect on inline classic scripts.","quote":"Warning: This attribute must not be used if the src attribute is absent (i.e., for inline scripts) for classic scripts, in this case it would have no effect.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"defer has no effect on inline scripts.","quote":"Warning: This attribute must not be used if the src attribute is absent (i.e., for inline scripts), in this case it would have no effect.","verified":true,"document":"mdn-script-element"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Module scripts are deferred by default; with async they and their dependencies are fetched in parallel and run as soon as available; defer does nothing for modules.","quote":"Otherwise, the module script and its dependencies will be fetched in parallel to parsing and evaluated when the page has finished parsing. (The defer attribute has no effect on module scripts.)","verified":true,"document":"whatwg-html-scripting"},{"url":"https://web.dev/learn/performance/optimize-resource-loading","claim":"Module scripts, including inline ones, are deferred automatically.","quote":"async and defer allow external scripts to load without blocking the HTML parser while scripts (including inline scripts) with type=\"module\" are deferred automatically.","verified":true,"document":"webdev-optimize-resource-loading"},{"url":"https://web.dev/learn/performance/optimize-resource-loading","claim":"web.dev's resource-loading page describes deferred execution as happening at the same time as DOMContentLoaded, which is less precise than MDN's 'before'.","quote":"Scripts loaded with async are parsed and executed immediately once downloaded, while scripts loaded with defer are executed when HTML document parsing is finished—this occurs at the same time as the browser's DOMContentLoaded event.","verified":true,"document":"webdev-optimize-resource-loading"}]
judge unverified quotes
0
judge verdicts
[{"parallel-download":{"status":"met","support":["E1","E2","E3"]},"defer-timing":{"status":"met","support":["E2","E4","E5"]},"async-timing":{"status":"met","support":["E3","E7"]},"ordering":{"status":"met","support":["E6","E7"]},"inline-ignored":{"status":"met","support":["E11","E12"]},"modules":{"status":"met","support":["E13","E14"]}},{"parallel-download":{"status":"met","support":["E2","E3","E1"]},"defer-timing":{"status":"met","support":["E4","E2"]},"async-timing":{"status":"met","support":["E3"]},"ordering":{"status":"met","support":["E6","E7"]},"inline-ignored":{"status":"met","support":["E11","E12"]},"modules":{"status":"met","support":["E13","E14"]}},{"parallel-download":{"status":"met","support":["E1","E2","E3"]},"defer-timing":{"status":"met","support":["E4","E5"]},"async-timing":{"status":"met","support":["E3","E7"]},"ordering":{"status":"met","support":["E6","E7"]},"inline-ignored":{"status":"met","support":["E11","E12"]},"modules":{"status":"met","support":["E13","E14"]}}]
judge failures
0
grounded weight share
1

Final answer

For an external classic script, both async and defer let the browser download the script while it keeps parsing the HTML. The difference is when the script runs. A defer script waits until parsing has finished and runs before DOMContentLoaded, and several defer scripts run in the order they appear in the document. An async script runs as soon as it has downloaded, in no guaranteed order, and its timing relative to DOMContentLoaded is not fixed. Neither attribute does anything on an inline classic script. Module scripts are deferred by default, so defer does nothing on them, while async works on both external and inline modules.

1. A plain classic script (<script src> with neither attribute)

With neither attribute, "the script is fetched and evaluated immediately, blocking parsing until these are both complete" (https://html.spec.whatwg.org/multipage/scripting.html). MDN says the same: such a script "blocks parsing, not rendering" (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script).

2. defer (external classic scripts)

3. async (external classic scripts)

  • Parsing: the script is "fetched in parallel to parsing and evaluated as soon as it is available (potentially before parsing completes)" (https://html.spec.whatwg.org/multipage/scripting.html). So the download doesn't block the parser, but running the script can. web.dev notes that async scripts "can interrupt DOM building if they finish downloading while the parser is still at work" (https://web.dev/articles/efficiently-load-third-party-javascript).
  • Order: none is guaranteed. The spec puts async scripts in a "set of scripts that will execute as soon as possible", which is a set, not an ordered list, and each one runs when it is ready (spec URL above). web.dev says async scripts are likely to run out of document order.
  • DOMContentLoaded: the event "doesn't wait for … async scripts" (MDN event page). An async script can run before the event, if it is ready while parsing is still going on, or after it. MDN itself lists <script async> as a case where your code may run after DOMContentLoaded has already fired.
  • Both attributes set: async wins. Adding defer only gives older browsers that support defer but not async a fallback (spec). MDN says the element "will act as if only the async attribute is specified".

4. Inline classic scripts

In the spec's attribute table, async and defer are not allowed on inline classic scripts. The async/defer handling only applies when the type is "classic" and the element has a src, or when the type is "module". Any other script is executed immediately, unless a style sheet that blocks scripts makes it wait (spec URL). MDN warns that using either attribute without src "would have no effect". web.dev adds that inline scripts "block the parser until the script is parsed and executed" (https://web.dev/learn/performance/optimize-resource-loading).

5. type="module"

Modules are deferred by default. Without async, a module and its dependencies are fetched while parsing continues and run when parsing has finished, and "The defer attribute has no effect on module scripts" (spec). They go in the same ordered "finished parsing" list as defer scripts, so they also run before DOMContentLoaded (MDN event page). With async, "the module script and all its dependencies will be fetched in parallel to parsing, and the module script will be evaluated as soon as it is available". Unlike classic scripts, async is allowed on inline module scripts as well (spec table). web.dev also says inline module scripts are deferred automatically.

Where the sources disagree

  • Async and DOMContentLoaded. A note on MDN's event page says DOMContentLoaded "is fired after the execution of these scripts, but before the execution of scripts with async". The spec says async scripts run "potentially before parsing completes", and web.dev agrees they can interrupt parsing. Both mean an async script can run before DOMContentLoaded. The same MDN page elsewhere says only that the event doesn't wait for async scripts. I'd go with the spec: there is no fixed order between async scripts and DOMContentLoaded.
  • "Same time" or "before". web.dev's resource-loading page says deferred scripts run when parsing finishes, which "occurs at the same time as the browser's DOMContentLoaded event". MDN and the other web.dev article are more precise: deferred scripts run before the event, and the event waits for them.
  • What I couldn't fetch. The spec's parsing chapter ("The end"), which spells out the steps for running deferred scripts and then firing DOMContentLoaded, returned 404 at https://html.spec.whatwg.org/multipage/parsing.html. The "before DOMContentLoaded" claim therefore rests on MDN and web.dev, plus the scripting chapter's description of the parser-run list.
{"evidence": [
{"claim": "With neither async nor defer, a classic external script is fetched and run immediately, blocking the parser.", "quote": "If neither attribute is present, then the script is fetched and evaluated immediately, blocking parsing until these are both complete.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "A deferred external classic script is fetched in parallel and runs after parsing finishes.", "quote": "If the async attribute is not present but the defer attribute is present, then the classic script will be fetched in parallel and evaluated when the page has finished parsing.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "An async external classic script is fetched in parallel and runs as soon as it is available, possibly before parsing completes.", "quote": "For external classic scripts, if the async attribute is present, then the classic script will be fetched in parallel to parsing and evaluated as soon as it is available (potentially before parsing completes).", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Deferred scripts run after parsing but before DOMContentLoaded.", "quote": "This Boolean attribute is set to indicate to a browser that the script is meant to be executed after the document has been parsed, but before firing DOMContentLoaded event.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "Deferred scripts hold back DOMContentLoaded until they finish.", "quote": "Scripts with the defer attribute will prevent the DOMContentLoaded event from firing until the script has loaded and finished evaluating.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "Deferred scripts run in document order.", "quote": "Scripts with the defer attribute will execute in the order in which they appear in the document.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "Async scripts likely run out of document order and can interrupt parsing.", "quote": "This means it's possible (and likely) that async scripts won't run in the order in which they appear in the HTML.", "url": "https://web.dev/articles/efficiently-load-third-party-javascript"},
{"claim": "DOMContentLoaded waits for parsing and deferred/module scripts but not for async scripts.", "quote": "It doesn't wait for other things like images, subframes, and async scripts to finish loading.", "url": "https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event"},
{"claim": "MDN's DOMContentLoaded note says async scripts run after the event, which conflicts with the spec's 'potentially before parsing completes'.", "quote": "DOMContentLoaded is fired after the execution of these scripts, but before the execution of scripts with async.", "url": "https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event"},
{"claim": "If both attributes are set, async takes precedence and defer is only a legacy fallback.", "quote": "The defer attribute may be specified even if the async attribute is specified, to cause legacy web browsers that only support defer (and not async) to fall back to the defer behavior instead of the blocking behavior that is the default.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "async has no effect on inline classic scripts.", "quote": "Warning: This attribute must not be used if the src attribute is absent (i.e., for inline scripts) for classic scripts, in this case it would have no effect.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "defer has no effect on inline scripts.", "quote": "Warning: This attribute must not be used if the src attribute is absent (i.e., for inline scripts), in this case it would have no effect.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "Module scripts are deferred by default; with async they and their dependencies are fetched in parallel and run as soon as available; defer does nothing for modules.", "quote": "Otherwise, the module script and its dependencies will be fetched in parallel to parsing and evaluated when the page has finished parsing. (The defer attribute has no effect on module scripts.)", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Module scripts, including inline ones, are deferred automatically.", "quote": "async and defer allow external scripts to load without blocking the HTML parser while scripts (including inline scripts) with type=\"module\" are deferred automatically.", "url": "https://web.dev/learn/performance/optimize-resource-loading"},
{"claim": "web.dev's resource-loading page describes deferred execution as happening at the same time as DOMContentLoaded, which is less precise than MDN's 'before'.", "quote": "Scripts loaded with async are parsed and executed immediately once downloaded, while scripts loaded with defer are executed when HTML document parsing is finished—this occurs at the same time as the browser's DOMContentLoaded event.", "url": "https://web.dev/learn/performance/optimize-resource-loading"}
]}

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.