ScaffoldBench

RunSonnet 5.5repeat 2

Propose-select on async vs defer on <script>

claude-sonnet-5-5__propose-select__research-script-loading__r2

Score

0.938

wrong answer
design ran as designed
Cost
$0.570
+ $0.101 judging
Latency
1m 14s
setup 0.0s, grading 5.2s
Turns
45
Tool calls
40
Subagents
3
Tokens in / out
330k / 23k
247,969 read from cache

Did the design run?

every check held
  • Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
  • No API failures: heldharness.apiNo model call failed at the API.
  • Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
  • No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
  • Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
  • No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
  • Builtin tools: heldharness.builtin-toolsA check recorded with the run.
  • Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
  • No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
  • Judge answered: heldharness.judgeAt least one grading call returned a usable verdict.
  • All three proposers ran: heldpropose-select.proposalsThree proposers ran in parallel, each from its own angle, and none was starved while another ran.
  • The selector chose: heldpropose-select.selectionThe selector picked a proposal by a valid number, read from its schema reply and not a default.
  • Each call was shown the right tools: heldpropose-select.toolsEach proposer was shown the read-only tools, the selector none, and the executor the full toolset.

Phases

Proposal 1 (policy-first): 0.0s to 42.9s, finished, 14 tool calls, $0.151
Proposal 2 (state-first): 0.0s to 42.0s, finished, 13 tool calls, $0.168
Proposal 3 (outcome-first): 0.0s to 39.7s, finished, 13 tool calls, $0.137
Select: 42.9s to 1m 14s, finished, 0 tool calls, $0.114

Drawn from the phase records: timing and cost per call, no tool calls or messages.

PhaseDurationCostTurnsToolsModel
Proposal 1 (policy-first)42.9s$0.1511514claude-sonnet-5-5
Proposal 2 (state-first)42.0s$0.1681413claude-sonnet-5-5
Proposal 3 (outcome-first)39.7s$0.1371413claude-sonnet-5-5
Select31.1s$0.11420claude-sonnet-5-5

The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.

Runs you can replay

Record this cell yourself: npm run bench -- run --strategies propose-select --tasks research-script-loading --models claude-sonnet-5-5 --repeats 1

Score breakdown

  • passStates that both async and defer let a classic script be fetched in parallel with HTML parsing, so the fetch does not block the parser, unlike a classic script with src and neither attribute.
    1.00weight 0.13

    met, backed by verified evidence E1, E2, E3. “`async` and `defer` both make an external script download in parallel with HTML parsing.”

  • passStates that a defer script executes only after the document has finished parsing, and before DOMContentLoaded fires.
    1.00weight 0.19

    met, backed by verified evidence E3, E9. “`defer` runs it after parsing finishes, in document order, before `DOMContentLoaded`.”

  • passStates that an async script executes as soon as it is available, interrupting parsing if parsing is still in progress, so it may run before or after DOMContentLoaded.
    1.00weight 0.19

    met, backed by verified evidence E2, E10. “An async script that arrives early can run before parsing ends, and so before `DOMContentLoaded`. A slow one runs after it.”

  • passStates that multiple defer scripts execute in document order, while async scripts execute in the order their downloads complete, which is not predictable.
    1.00weight 0.19

    met, backed by verified evidence E12, E13. “`defer`: document order.** MDN says "Scripts with the defer attribute will execute in the order in which they appear in the document."”

  • partialStates that async and defer have no effect on a classic inline script (a script element without a src attribute).
    0.50weight 0.13

    partial, backed by verified evidence E14. “MDN says that for inline classic scripts the attribute "would have no effect."”

  • passStates that module scripts are deferred by default, so the defer attribute has no effect on them, while the async attribute does apply to module scripts.
    1.00weight 0.19

    met, backed by verified evidence E4, E16. “The module and all its dependencies are fetched in parallel and run as soon as they are available, possibly before parsing completes.”

Penalties

No penalties were applied.

Scorer metadata

corpus
{"version":2,"snapshot":"2026-10-01","hash":"a5d703ba820ee9c957565ac15b9f98daeac8212372d73d7658a7a6bb2bbc1978","documents":6}
opened documents
["mdn-script-element","whatwg-html-scripting","mdn-domcontentloaded"]
opened by role
{"secondary":["mdn-script-element","mdn-domcontentloaded"],"primary":["whatwg-html-scripting"]}
displayed share
{"mdn-script-element":0.8439,"whatwg-html-scripting":0.4616,"mdn-domcontentloaded":1,"webdev-third-party-javascript":0.0309,"webdev-optimize-resource-loading":0.0337,"mdn-js-modules":0.0055}
cited count
3
verified count
3
hallucinated count
0
citation integrity
1
hallucinated urls
[]
evidence block
true
evidence block repaired
false
evidence count
18
verified quote count
18
verified quote rate
1
evidence
[{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"A plain external classic script blocks parsing while it is fetched and evaluated","quote":"If neither attribute is present, then the script is fetched and evaluated immediately, blocking parsing until these are both complete.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"async classic scripts are fetched in parallel and run as soon as available, possibly before parsing completes","quote":"then the classic script will be fetched in parallel to parsing and evaluated as soon as it is available (potentially before parsing completes).","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"defer classic scripts are fetched in parallel and run once parsing has finished","quote":"If the async attribute is not present but the defer attribute is present, then the classic script will be fetched in parallel and evaluated when the page has finished parsing.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Module scripts are deferred by default and defer has no effect on them","quote":"(The defer attribute has no effect on module scripts.)","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Module scripts wait for parsing to finish regardless of where they appear in the document","quote":"no matter where this script element appears in the document, it will not be evaluated until both document parsing has complete","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Parser-inserted deferred and module scripts share the same post-parsing ordered list","quote":"Otherwise, if el has a defer attribute or el's type is \"module\":","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Setting both async and defer is allowed so legacy defer-only browsers fall back to deferred behaviour","quote":"The defer attribute may be specified even if the async attribute is specified, to cause legacy web browsers that only support defer (and not async) to fall back to the defer behavior instead of the blocking behavior that is the default.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"With both attributes present, async takes precedence","quote":"If the attribute is specified with the async attribute, the element will act as if only the async attribute is specified.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"defer scripts run before DOMContentLoaded and hold it up until they finish","quote":"Scripts with the defer attribute will prevent the DOMContentLoaded event from firing until the script has loaded and finished evaluating.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event","claim":"DOMContentLoaded waits for deferred and module scripts but not async scripts","quote":"It doesn't wait for other things like images, subframes, and async scripts to finish loading.","verified":true,"document":"mdn-domcontentloaded"},{"url":"https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event","claim":"MDN's event page says DOMContentLoaded fires before async scripts execute (the spec instead says async may run before parsing completes)","quote":"DOMContentLoaded is fired after the execution of these scripts, but before the execution of scripts with async.","verified":true,"document":"mdn-domcontentloaded"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"Deferred scripts execute in document order","quote":"Scripts with the defer attribute will execute in the order in which they appear in the document.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"async scripts have no ordering guarantee","quote":"You get no guarantee that scripts will run in any specific order.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"async has no effect on inline classic scripts","quote":"This attribute must not be used if the src attribute is absent (i.e., for inline scripts) for classic scripts, in this case it would have no effect.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"Inline non-module scripts run immediately during parsing","quote":"Scripts without async, defer or type=\"module\" attributes, as well as inline scripts without the type=\"module\" attribute, are fetched and executed immediately before the browser continues to parse the page.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"async on module scripts fetches the module and its dependencies in parallel and runs it as soon as available","quote":"For module scripts, if the async attribute is present then the scripts and all their dependencies will be fetched in parallel to parsing and evaluated as soon as they are available.","verified":true,"document":"mdn-script-element"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Parser-inserted scripts have force async false, while dynamically created scripts default to force async true","quote":"A script element has a force async boolean, initially true. It is set to false by the HTML parser and the XML parser on script elements they insert, and when the element gets an async content attribute added.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Inline module scripts are not run synchronously; they are queued as a task even with no dependencies","quote":"Queueing a task here means that, even if the inline module script has no dependencies or synchronously results in a parse error, we won't proceed to execute the script element synchronously.","verified":true,"document":"whatwg-html-scripting"}]
judge unverified quotes
0
judge verdicts
[{"parallel-download":{"status":"met","support":["E1","E2","E3"]},"defer-timing":{"status":"met","support":["E3","E9"]},"async-timing":{"status":"met","support":["E2","E10"]},"ordering":{"status":"met","support":["E12","E13"]},"inline-ignored":{"status":"partial","support":["E14"]},"modules":{"status":"met","support":["E4","E16"]}},{"parallel-download":{"status":"met","support":["E1","E2","E3"]},"defer-timing":{"status":"met","support":["E3","E9"]},"async-timing":{"status":"met","support":["E2","E10"]},"ordering":{"status":"met","support":["E12","E13"]},"inline-ignored":{"status":"partial","support":["E14"]},"modules":{"status":"met","support":["E4","E16"]}},{"parallel-download":{"status":"met","support":["E1","E2","E3"]},"defer-timing":{"status":"met","support":["E3","E9"]},"async-timing":{"status":"met","support":["E2","E10"]},"ordering":{"status":"met","support":["E12","E13"]},"inline-ignored":{"status":"partial","support":["E14"]},"modules":{"status":"met","support":["E4","E16"]}}]
judge failures
0
grounded weight share
1

Final answer

async and defer both make an external script download in parallel with HTML parsing. They differ in when the script runs and whether order is guaranteed. async runs the script as soon as it has arrived, in no guaranteed order. defer runs it after parsing finishes, in document order, before DOMContentLoaded.

What each does

No attribute. A plain external classic script is fetched and run immediately, and the parser stops until both are done. The spec says: "If neither attribute is present, then the script is fetched and evaluated immediately, blocking parsing until these are both complete." (https://html.spec.whatwg.org/multipage/scripting.html)

async. For an external classic script, "the classic script will be fetched in parallel to parsing and evaluated as soon as it is available (potentially before parsing completes)." (https://html.spec.whatwg.org/multipage/scripting.html)

defer. The classic script "will be fetched in parallel and evaluated when the page has finished parsing." (https://html.spec.whatwg.org/multipage/scripting.html)

Both together. async wins. MDN says "the element will act as if only the async attribute is specified." The spec says authors may still write both so that legacy browsers that only know defer fall back to deferred behaviour instead of blocking. (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script and https://html.spec.whatwg.org/multipage/scripting.html)

Timing relative to DOMContentLoaded

  • defer scripts hold up DOMContentLoaded. MDN's event page says it fires once the document is parsed "and all deferred scripts (<script defer src="…"> and <script type="module">) have downloaded and executed." It also says the event "doesn't wait for other things like images, subframes, and async scripts to finish loading." (https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event)
  • The sequence for deferred scripts. document.readyState becomes "interactive" when parsing completes, before the deferred and module scripts run. DOMContentLoaded then fires after they finish. (same MDN event page)
  • async scripts have no fixed position. DOMContentLoaded doesn't wait for them. An async script that arrives early can run before parsing ends, and so before DOMContentLoaded. A slow one runs after it. Code in an async script should check document.readyState instead of assuming either case.
  • A difference in wording between the sources. The spec says an async script may run "potentially before parsing completes". A note on MDN's event page says DOMContentLoaded fires "before the execution of scripts with async." The spec's wording is the broader one. Treat the MDN note as describing the common case, not a guarantee.

Execution order with several scripts

Inline scripts

async and defer apply only to external classic scripts.

type="module" scripts

  • Deferred by default. The spec says "Otherwise, the module script and its dependencies will be fetched in parallel to parsing and evaluated when the page has finished parsing. (The defer attribute has no effect on module scripts.)" (https://html.spec.whatwg.org/multipage/scripting.html)
  • Same ordered list as defer. In the spec's algorithm, a parser-inserted script goes into the post-parsing ordered list if it "has a defer attribute or el's type is "module"". Module scripts therefore run in document order, before DOMContentLoaded, alongside deferred classic scripts. Placement in the document doesn't change this.
  • async on modules. The module and all its dependencies are fetched in parallel and run as soon as they are available, possibly before parsing completes. There is no ordering guarantee, as with async classic scripts.
  • Inline modules are the exception to the inline rule. The spec's attribute table allows async on inline module scripts. Without async, an inline module is deferred. With async, it goes into the as-soon-as-possible set. Unlike an inline classic script, it never runs synchronously at parse time. The spec says it always waits for a queued task, even when it has no dependencies.
{"evidence": [
{"claim": "A plain external classic script blocks parsing while it is fetched and evaluated", "quote": "If neither attribute is present, then the script is fetched and evaluated immediately, blocking parsing until these are both complete.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "async classic scripts are fetched in parallel and run as soon as available, possibly before parsing completes", "quote": "then the classic script will be fetched in parallel to parsing and evaluated as soon as it is available (potentially before parsing completes).", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "defer classic scripts are fetched in parallel and run once parsing has finished", "quote": "If the async attribute is not present but the defer attribute is present, then the classic script will be fetched in parallel and evaluated when the page has finished parsing.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Module scripts are deferred by default and defer has no effect on them", "quote": "(The defer attribute has no effect on module scripts.)", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Module scripts wait for parsing to finish regardless of where they appear in the document", "quote": "no matter where this script element appears in the document, it will not be evaluated until both document parsing has complete", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Parser-inserted deferred and module scripts share the same post-parsing ordered list", "quote": "Otherwise, if el has a defer attribute or el's type is \"module\":", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Setting both async and defer is allowed so legacy defer-only browsers fall back to deferred behaviour", "quote": "The defer attribute may be specified even if the async attribute is specified, to cause legacy web browsers that only support defer (and not async) to fall back to the defer behavior instead of the blocking behavior that is the default.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "With both attributes present, async takes precedence", "quote": "If the attribute is specified with the async attribute, the element will act as if only the async attribute is specified.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "defer scripts run before DOMContentLoaded and hold it up until they finish", "quote": "Scripts with the defer attribute will prevent the DOMContentLoaded event from firing until the script has loaded and finished evaluating.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "DOMContentLoaded waits for deferred and module scripts but not async scripts", "quote": "It doesn't wait for other things like images, subframes, and async scripts to finish loading.", "url": "https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event"},
{"claim": "MDN's event page says DOMContentLoaded fires before async scripts execute (the spec instead says async may run before parsing completes)", "quote": "DOMContentLoaded is fired after the execution of these scripts, but before the execution of scripts with async.", "url": "https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event"},
{"claim": "Deferred scripts execute in document order", "quote": "Scripts with the defer attribute will execute in the order in which they appear in the document.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "async scripts have no ordering guarantee", "quote": "You get no guarantee that scripts will run in any specific order.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "async has no effect on inline classic scripts", "quote": "This attribute must not be used if the src attribute is absent (i.e., for inline scripts) for classic scripts, in this case it would have no effect.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "Inline non-module scripts run immediately during parsing", "quote": "Scripts without async, defer or type=\"module\" attributes, as well as inline scripts without the type=\"module\" attribute, are fetched and executed immediately before the browser continues to parse the page.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "async on module scripts fetches the module and its dependencies in parallel and runs it as soon as available", "quote": "For module scripts, if the async attribute is present then the scripts and all their dependencies will be fetched in parallel to parsing and evaluated as soon as they are available.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "Parser-inserted scripts have force async false, while dynamically created scripts default to force async true", "quote": "A script element has a force async boolean, initially true. It is set to false by the HTML parser and the XML parser on script elements they insert, and when the element gets an async content attribute added.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Inline module scripts are not run synchronously; they are queued as a task even with no dependencies", "quote": "Queueing a task here means that, even if the inline module script has no dependencies or synchronously results in a parse error, we won't proceed to execute the script element synchronously.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"}
]}

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.