ScaffoldBench

RunSonnet 5.5repeat 2

Gated-execute on async vs defer on <script>

claude-sonnet-5-5__gated-execute__research-script-loading__r2

Score

0.900

hallucinated citation
design ran as designed
Cost
$0.135
+ $0.084 judging
Latency
36.2s
setup 0.0s, grading 4.5s
Turns
15
Tool calls
14
Subagents
0
Tokens in / out
109k / 5.0k
91,847 read from cache

Did the design run?

every check held
  • Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
  • No API failures: heldharness.apiNo model call failed at the API.
  • Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
  • No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
  • Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
  • No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
  • Builtin tools: heldharness.builtin-toolsA check recorded with the run.
  • Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
  • No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
  • Judge answered: heldharness.judgeAt least one grading call returned a usable verdict.
  • Every state change was reviewed: heldgated-execute.reviewedEvery call that changed the world had been allowed by a review before it ran, counted from the world’s own journal.
  • Every decision was read: heldgated-execute.decisionsEvery review that ran to its end gave its decision through the schema; an unreadable one is a denial.
  • Each call was shown the right tools: heldgated-execute.toolsThe executor was shown the task’s full toolset and each reviewer the read-only tools.

Phases

Execute: 0.0s to 36.2s, finished, 14 tool calls, $0.135

Drawn from the phase records: timing and cost per call, no tool calls or messages.

PhaseDurationCostTurnsToolsModel
Execute36.2s$0.1351514claude-sonnet-5-5

The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.

Runs you can replay

Record this cell yourself: npm run bench -- run --strategies gated-execute --tasks research-script-loading --models claude-sonnet-5-5 --repeats 1

Score breakdown

  • passStates that both async and defer let a classic script be fetched in parallel with HTML parsing, so the fetch does not block the parser, unlike a classic script with src and neither attribute.
    1.00weight 0.13

    met, backed by verified evidence E1, E2, E3. “`async` and `defer` both let an external script download while the HTML parser keeps going.”

  • passStates that a defer script executes only after the document has finished parsing, and before DOMContentLoaded fires.
    1.00weight 0.19

    met, backed by verified evidence E3, E7, E11. “after the document has been parsed, but before firing DOMContentLoaded event”

  • passStates that an async script executes as soon as it is available, interrupting parsing if parsing is still in progress, so it may run before or after DOMContentLoaded.
    1.00weight 0.19

    met, backed by verified evidence E2, E10. “An async script may run before parsing ends, or after DOMContentLoaded has already fired.”

  • passStates that multiple defer scripts execute in document order, while async scripts execute in the order their downloads complete, which is not predictable.
    1.00weight 0.19

    met, backed by verified evidence E6, E12. “There is no guarantee. web.dev says "async scripts may execute out-of-order"”

  • passStates that async and defer have no effect on a classic inline script (a script element without a src attribute).
    1.00weight 0.13

    met, backed by verified evidence E8. “Neither attribute does anything without `src`.”

  • passStates that module scripts are deferred by default, so the defer attribute has no effect on them, while the async attribute does apply to module scripts.
    1.00weight 0.19

    met, backed by verified evidence E4. “(The defer attribute has no effect on module scripts.)”

Penalties

  • hallucinated citationCited pages that were never opened×0.9

    1 of 5 cited URLs were never opened: https://web.dev/articles/efficiently-load-third-party-javascript

Scorer metadata

corpus
{"version":2,"snapshot":"2026-10-01","hash":"a5d703ba820ee9c957565ac15b9f98daeac8212372d73d7658a7a6bb2bbc1978","documents":6}
opened documents
["whatwg-html-scripting","mdn-script-element","mdn-domcontentloaded","webdev-optimize-resource-loading"]
opened by role
{"primary":["whatwg-html-scripting"],"secondary":["mdn-script-element","mdn-domcontentloaded","webdev-optimize-resource-loading"]}
displayed share
{"mdn-script-element":0.444,"whatwg-html-scripting":0.2305,"webdev-optimize-resource-loading":0.0152,"mdn-domcontentloaded":1,"webdev-third-party-javascript":0.0303}
cited count
5
verified count
4
hallucinated count
1
citation integrity
0.8 (80%)
hallucinated urls
["https://web.dev/articles/efficiently-load-third-party-javascript"]
evidence block
true
evidence block repaired
false
evidence count
12
verified quote count
12
verified quote rate
1
evidence
[{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"A classic external script with neither attribute blocks parsing while it is fetched and run.","quote":"If neither attribute is present, then the script is fetched and evaluated immediately, blocking parsing until these are both complete.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"async classic scripts are fetched in parallel and run as soon as available, possibly before parsing completes.","quote":"For external classic scripts, if the async attribute is present, then the classic script will be fetched in parallel to parsing and evaluated as soon as it is available (potentially before parsing completes).","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"defer classic scripts are fetched in parallel and evaluated after parsing finishes.","quote":"If the async attribute is not present but the defer attribute is present, then the classic script will be fetched in parallel and evaluated when the page has finished parsing.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Module scripts are deferred by default; defer has no effect on them; async makes them run as soon as available.","quote":"Otherwise, the module script and its dependencies will be fetched in parallel to parsing and evaluated when the page has finished parsing. (The defer attribute has no effect on module scripts.)","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"defer may be combined with async as a fallback for legacy browsers.","quote":"The defer attribute may be specified even if the async attribute is specified, to cause legacy web browsers that only support defer (and not async) to fall back to the defer behavior instead of the blocking behavior that is the default.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"Deferred scripts execute in document order and delay DOMContentLoaded; defer is ignored for inline scripts and module scripts.","quote":"Scripts with the defer attribute will execute in the order in which they appear in the document.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"defer scripts block DOMContentLoaded until they have loaded and run.","quote":"Scripts with the defer attribute will prevent the DOMContentLoaded event from firing until the script has loaded and finished evaluating.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"async on inline classic scripts has no effect.","quote":"This attribute must not be used if the src attribute is absent (i.e., for inline scripts) for classic scripts, in this case it would have no effect.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"If both async and defer are present, async wins.","quote":"If the attribute is specified with the defer attribute, the element will act as if only the async attribute is specified.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event","claim":"DOMContentLoaded waits for deferred and module scripts but not async scripts.","quote":"It doesn't wait for other things like images, subframes, and async scripts to finish loading.","verified":true,"document":"mdn-domcontentloaded"},{"url":"https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event","claim":"readyState is interactive during deferred/module script execution, and DOMContentLoaded fires after them.","quote":"DOMContentLoaded is fired after the execution of these scripts, but before the execution of scripts with async.","verified":true,"document":"mdn-domcontentloaded"},{"url":"https://web.dev/learn/performance/optimize-resource-loading","claim":"async scripts have no ordering guarantee while defer scripts keep markup order.","quote":"Additionally, async scripts may execute out-of-order, while defer scripts are executed in the order in which they appear in the markup.","verified":true,"document":"webdev-optimize-resource-loading"}]
judge unverified quotes
0
judge verdicts
[{"parallel-download":{"status":"met","support":["E1","E2","E3"]},"defer-timing":{"status":"met","support":["E3","E7","E11"]},"async-timing":{"status":"met","support":["E2","E10"]},"ordering":{"status":"met","support":["E6","E12"]},"inline-ignored":{"status":"met","support":["E8"]},"modules":{"status":"met","support":["E4"]}},{"parallel-download":{"status":"met","support":["E1","E2","E3"]},"defer-timing":{"status":"met","support":["E3","E7","E11"]},"async-timing":{"status":"met","support":["E2","E10"]},"ordering":{"status":"met","support":["E12","E6"]},"inline-ignored":{"status":"met","support":["E8"]},"modules":{"status":"met","support":["E4"]}},{"parallel-download":{"status":"met","support":["E1","E2","E3"]},"defer-timing":{"status":"met","support":["E3","E7","E11"]},"async-timing":{"status":"met","support":["E2","E10"]},"ordering":{"status":"met","support":["E12","E6"]},"inline-ignored":{"status":"met","support":["E8"]},"modules":{"status":"met","support":["E4"]}}]
judge failures
0
grounded weight share
1

Final answer

async and defer both let an external script download while the HTML parser keeps going. They differ in when the script runs and whether the order of several scripts is kept.

Baseline. A classic <script src> with neither attribute is fetched and run immediately, and parsing stops until both are done. The HTML spec says "If neither attribute is present, then the script is fetched and evaluated immediately, blocking parsing until these are both complete." (https://html.spec.whatwg.org/multipage/scripting.html)

async (classic external script)

  • Parsing: The script is "fetched in parallel to parsing and evaluated as soon as it is available (potentially before parsing completes)" (spec). Download doesn't block the parser, but execution can interrupt it. web.dev notes that async scripts "can interrupt DOM building if they finish downloading while the parser is still at work" (https://web.dev/articles/efficiently-load-third-party-javascript). I only saw that in a search snippet, so it is not backed by an opened page.
  • DOMContentLoaded: There is no fixed relationship. MDN says DOMContentLoaded "doesn't wait for other things like images, subframes, and async scripts" (https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event). An async script may run before parsing ends, or after DOMContentLoaded has already fired. MDN's note that DOMContentLoaded fires "before the execution of scripts with async" reads more strictly than the spec's "potentially before parsing completes". I'd treat async timing as unpredictable. Any async script's setup code should check document.readyState before waiting for DOMContentLoaded.
  • Order: There is no guarantee. web.dev says "async scripts may execute out-of-order" (https://web.dev/learn/performance/optimize-resource-loading). In the spec's processing model, an async script is added to a "set" of scripts that will execute as soon as possible, and each runs when its own result is ready.

defer (classic external script)

  • Parsing: The script is "fetched in parallel and evaluated when the page has finished parsing" (spec).
  • DOMContentLoaded: MDN says defer scripts run "after the document has been parsed, but before firing DOMContentLoaded event", and that they "prevent the DOMContentLoaded event from firing until the script has loaded and finished evaluating" (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script). document.readyState is "interactive" while they run (MDN DOMContentLoaded page).
  • Order: Document order is guaranteed. MDN says "Scripts with the defer attribute will execute in the order in which they appear in the document." In the spec, deferred scripts go into an ordered "list of scripts that will execute when the document has finished parsing."

Both attributes on one element. MDN says that if async and defer are both present, the element acts as if only async is specified. The spec says defer may still be added alongside async so that legacy browsers that only know defer fall back to deferring instead of blocking.

Inline classic scripts. Neither attribute does anything without src. MDN warns that for inline scripts they "would have no effect". The spec's algorithm only applies async and defer handling to classic scripts that have a src attribute, or to module scripts. Any other inline classic script is, in the spec's words, immediately executed. It runs synchronously as the parser reaches it (unless a stylesheet is still blocking scripts).

type="module" scripts

  • Default behavior: Modules are deferred by default. The spec says "the module script and its dependencies will be fetched in parallel to parsing and evaluated when the page has finished parsing. (The defer attribute has no effect on module scripts.)" They are put in the same ordered list as defer scripts. They run in document order, before DOMContentLoaded, and DOMContentLoaded waits for them (MDN DOMContentLoaded page).
  • With async: The module and all its dependencies are fetched in parallel and evaluated as soon as they are available, possibly before parsing completes. Order is then not guaranteed, as with classic async.
  • Inline modules: Unlike inline classic scripts, these follow the same rules as external modules. The spec's algorithm treats any module (with or without src) via the async or deferred-list branches, so an inline module is deferred by default and async applies to it.
  • Cross-origin: Unlike classic scripts, module scripts require CORS for cross-origin fetching (spec).

Dynamically inserted scripts. I didn't research this in depth, but the spec's algorithm has a branch for scripts that aren't parser-inserted. Those go into an "in order as soon as possible" list and run in insertion order. The spec also tracks a "force async" flag, so scripts created from JavaScript behave as async unless you set async = false.

{"evidence": [
{"claim": "A classic external script with neither attribute blocks parsing while it is fetched and run.", "quote": "If neither attribute is present, then the script is fetched and evaluated immediately, blocking parsing until these are both complete.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "async classic scripts are fetched in parallel and run as soon as available, possibly before parsing completes.", "quote": "For external classic scripts, if the async attribute is present, then the classic script will be fetched in parallel to parsing and evaluated as soon as it is available (potentially before parsing completes).", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "defer classic scripts are fetched in parallel and evaluated after parsing finishes.", "quote": "If the async attribute is not present but the defer attribute is present, then the classic script will be fetched in parallel and evaluated when the page has finished parsing.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Module scripts are deferred by default; defer has no effect on them; async makes them run as soon as available.", "quote": "Otherwise, the module script and its dependencies will be fetched in parallel to parsing and evaluated when the page has finished parsing. (The defer attribute has no effect on module scripts.)", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "defer may be combined with async as a fallback for legacy browsers.", "quote": "The defer attribute may be specified even if the async attribute is specified, to cause legacy web browsers that only support defer (and not async) to fall back to the defer behavior instead of the blocking behavior that is the default.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Deferred scripts execute in document order and delay DOMContentLoaded; defer is ignored for inline scripts and module scripts.", "quote": "Scripts with the defer attribute will execute in the order in which they appear in the document.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "defer scripts block DOMContentLoaded until they have loaded and run.", "quote": "Scripts with the defer attribute will prevent the DOMContentLoaded event from firing until the script has loaded and finished evaluating.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "async on inline classic scripts has no effect.", "quote": "This attribute must not be used if the src attribute is absent (i.e., for inline scripts) for classic scripts, in this case it would have no effect.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "If both async and defer are present, async wins.", "quote": "If the attribute is specified with the defer attribute, the element will act as if only the async attribute is specified.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "DOMContentLoaded waits for deferred and module scripts but not async scripts.", "quote": "It doesn't wait for other things like images, subframes, and async scripts to finish loading.", "url": "https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event"},
{"claim": "readyState is interactive during deferred/module script execution, and DOMContentLoaded fires after them.", "quote": "DOMContentLoaded is fired after the execution of these scripts, but before the execution of scripts with async.", "url": "https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event"},
{"claim": "async scripts have no ordering guarantee while defer scripts keep markup order.", "quote": "Additionally, async scripts may execute out-of-order, while defer scripts are executed in the order in which they appear in the markup.", "url": "https://web.dev/learn/performance/optimize-resource-loading"}
]}

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.