ScaffoldBench

RunHaiku 4.5repeat 1

Plan-execute-verify on async vs defer on <script>

claude-haiku-4-5-20251001__plan-execute-verify__research-script-loading__r1

Score

1.000

passed
design ran as designed
Cost
$0.523
+ $0.090 judging
Latency
6m 28s
setup 0.0s, grading 4.9s
Turns
72
Tool calls
62
Subagents
0
Tokens in / out
904k / 38k
759,848 read from cache

Did the design run?

every check held
  • Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
  • No API failures: heldharness.apiNo model call failed at the API.
  • Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
  • No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
  • Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
  • No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
  • Builtin tools: heldharness.builtin-toolsA check recorded with the run.
  • Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
  • No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
  • Judge answered: heldharness.judgeAt least one grading call returned a usable verdict.
  • The plan was a plan: heldplan-execute-verify.planThe planner’s reply was read through its schema and held at least one step.
  • Every verdict was read: heldplan-execute-verify.verdictsEvery review round ended with a verdict read from the schema reply, not a default.
  • The loop did what its verdicts said: heldplan-execute-verify.loopEvery verdict that called for a revision got one, and nothing was revised without such a verdict.
  • Each call was shown the right tools: heldplan-execute-verify.toolsThe planner was shown no tools, the verifier the read-only ones, and the executor and reviser the full toolset.

Phases

Plan: 0.0s to 14.2s, finished, 0 tool calls, $0.013
Execute: 14.2s to 1m 8s, finished, 13 tool calls, $0.092
Verify: 1m 8s to 3m 3s, finished, 30 tool calls, $0.171
Revise (round 1): 3m 3s to 3m 31s, finished, 0 tool calls, $0.031
Verify revision 1: 3m 31s to 5m 43s, finished, 16 tool calls, $0.163
Revise (round 2): 5m 43s to 6m 28s, finished, 3 tool calls, $0.053

Drawn from the phase records: timing and cost per call, no tool calls or messages.

PhaseDurationCostTurnsToolsModel
Plan14.2s$0.01330claude-haiku-4-5-20251001
Execute53.7s$0.0921413claude-haiku-4-5-20251001
Verify1m 55s$0.1713230claude-haiku-4-5-20251001
Revise (round 1)28.5s$0.03110claude-haiku-4-5-20251001
Verify revision 12m 12s$0.1631816claude-haiku-4-5-20251001
Revise (round 2)45.3s$0.05343claude-haiku-4-5-20251001

The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.

Runs you can replay

Record this cell yourself: npm run bench -- run --strategies plan-execute-verify --tasks research-script-loading --models claude-haiku-4-5-20251001 --repeats 1

Score breakdown

  • passStates that both async and defer let a classic script be fetched in parallel with HTML parsing, so the fetch does not block the parser, unlike a classic script with src and neither attribute.
    1.00weight 0.13

    met, backed by verified evidence E12, E1, E11. “Both async and defer eliminate blocking during the fetch phase”

  • passStates that a defer script executes only after the document has finished parsing, and before DOMContentLoaded fires.
    1.00weight 0.19

    met, backed by verified evidence E9, E5. “deferred scripts execute before DOMContentLoaded fires, and DOMContentLoaded waits for all deferred scripts to complete”

  • passStates that an async script executes as soon as it is available, interrupting parsing if parsing is still in progress, so it may run before or after DOMContentLoaded.
    1.00weight 0.19

    met, backed by verified evidence E1, E5. “async scripts can execute either before or after DOMContentLoaded fires, depending on whether they finish downloading before or after parsing completes”

  • passStates that multiple defer scripts execute in document order, while async scripts execute in the order their downloads complete, which is not predictable.
    1.00weight 0.19

    met, backed by verified evidence E3, E4. “"scripts with a defer attribute will load in the order they appear on the page.”

  • passStates that async and defer have no effect on a classic inline script (a script element without a src attribute).
    1.00weight 0.13

    met, backed by verified evidence E6, E10. “This attribute must not be used if the src attribute is absent (i.e., for inline scripts), in this case it would have no effect”

  • passStates that module scripts are deferred by default, so the defer attribute has no effect on them, while the async attribute does apply to module scripts.
    1.00weight 0.19

    met, backed by verified evidence E15, E14. “"The defer attribute has no effect on module scripts — they defer by default"”

Penalties

No penalties were applied.

Scorer metadata

corpus
{"version":2,"snapshot":"2026-10-01","hash":"a5d703ba820ee9c957565ac15b9f98daeac8212372d73d7658a7a6bb2bbc1978","documents":6}
opened documents
["mdn-script-element","mdn-domcontentloaded","webdev-third-party-javascript","whatwg-html-scripting","webdev-optimize-resource-loading"]
opened by role
{"secondary":["mdn-script-element","mdn-domcontentloaded","webdev-third-party-javascript","webdev-optimize-resource-loading"],"primary":["whatwg-html-scripting"]}
displayed share
{"whatwg-html-scripting":0.2316,"webdev-third-party-javascript":1,"webdev-optimize-resource-loading":1,"mdn-script-element":1,"mdn-js-modules":0.0056,"mdn-domcontentloaded":1}
cited count
4
verified count
4
hallucinated count
0
citation integrity
1
hallucinated urls
[]
evidence block
true
evidence block repaired
false
evidence count
16
verified quote count
13
verified quote rate
0.8125 (81%)
evidence
[{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"async classic scripts are fetched in parallel to parsing and evaluated as soon as they are available","quote":"the classic script will be fetched in parallel to parsing and evaluated as soon as it is available","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"once an async script download completes, the script executes and blocks the page from rendering","quote":"once the download is complete, the script will execute, which blocks the page from rendering.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"async scripts execute as soon as download completes without guaranteeing execution order","quote":"scripts with an async attribute will execute as soon as the download is complete. This blocks the page and does not guarantee any specific execution order.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"you cannot rely on the order async scripts execute in","quote":"you can't rely on the order the scripts will load in.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event","claim":"DOMContentLoaded does not wait for async scripts to finish loading","quote":"The DOMContentLoaded event fires when the HTML document has been completely parsed, and all deferred scripts (<script defer src=\"…\"> and <script type=\"module\">) have downloaded and executed. It doesn't wait for other things like images, subframes, and async scripts to finish loading.","verified":true,"document":"mdn-domcontentloaded"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"async attribute must not be used on inline scripts (those without src attribute)","quote":"This attribute must not be used if the src attribute is absent (i.e., for inline scripts) for classic scripts, in this case it would have no effect.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"defer classic scripts are fetched in parallel and evaluated when page has finished parsing","quote":"the classic script will be fetched in parallel and evaluated when the page has finished parsing.","verified":false,"document":"mdn-script-element","failure":"the quote does not occur in the page"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"defer scripts execute in the order they appear in the document","quote":"scripts with a defer attribute will load in the order they appear on the page. They won't run until the page content has all loaded, which is useful if your scripts depend on the DOM being in place","verified":false,"document":"mdn-script-element","failure":"the quote does not occur in the page"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"defer scripts prevent DOMContentLoaded from firing until scripts have loaded and finished evaluating","quote":"Scripts with the defer attribute will prevent the DOMContentLoaded event from firing until the script has loaded and finished evaluating.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"defer attribute must not be used on inline scripts","quote":"This attribute must not be used if the src attribute is absent (i.e., for inline scripts), in this case it would have no effect.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"Scripts without async, defer or type=module are fetched and executed immediately before parsing continues","quote":"Scripts without async, defer or type=\"module\" attributes, as well as inline scripts without the type=\"module\" attribute, are fetched and executed immediately before the browser continues to parse the page.","verified":true,"document":"mdn-script-element"},{"url":"https://web.dev/learn/performance/optimize-resource-loading","claim":"async and defer allow external scripts to load without blocking the HTML parser","quote":"async and defer allow external scripts to load without blocking the HTML parser while scripts (including inline scripts) with type=\"module\" are deferred automatically.","verified":true,"document":"webdev-optimize-resource-loading"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Module scripts with async are fetched in parallel and evaluated as soon as available","quote":"For module scripts, if the async attribute is present, then the module script and all its dependencies will be fetched in parallel to parsing and evaluated as soon as it is available (potentially before parsing completes).","verified":false,"document":"whatwg-html-scripting","failure":"the quote does not occur in the page"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Module scripts without async are fetched in parallel and evaluated when page has finished parsing","quote":"the module script and its dependencies will be fetched in parallel to parsing and evaluated when the page has finished parsing.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"defer attribute has no effect on module scripts as they defer by default","quote":"The defer attribute has no effect on module scripts — they defer by default.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event","claim":"DOMContentLoaded waits for both defer scripts and module scripts to complete","quote":"The DOMContentLoaded event fires when the HTML document has been completely parsed, and all deferred scripts (<script defer src=\"…\"> and <script type=\"module\">) have downloaded and executed.","verified":true,"document":"mdn-domcontentloaded"}]
judge unverified quotes
0
judge verdicts
[{"parallel-download":{"status":"met","support":["E12","E1","E7","E11"]},"defer-timing":{"status":"met","support":["E7","E9","E5"]},"async-timing":{"status":"met","support":["E1","E5","E13"]},"ordering":{"status":"met","support":["E8","E3","E4"]},"inline-ignored":{"status":"met","support":["E6","E10"]},"modules":{"status":"met","support":["E15","E13","E14"]}},{"parallel-download":{"status":"met","support":["E12","E1","E7","E11"]},"defer-timing":{"status":"met","support":["E9","E7","E16"]},"async-timing":{"status":"met","support":["E1","E5","E13"]},"ordering":{"status":"met","support":["E8","E3","E4"]},"inline-ignored":{"status":"met","support":["E6","E10"]},"modules":{"status":"met","support":["E15","E13","E14"]}},{"parallel-download":{"status":"met","support":["E1","E7","E12","E11"]},"defer-timing":{"status":"met","support":["E7","E9","E5"]},"async-timing":{"status":"met","support":["E1","E2","E5"]},"ordering":{"status":"met","support":["E8","E3","E4"]},"inline-ignored":{"status":"met","support":["E6","E10"]},"modules":{"status":"met","support":["E15","E13","E14"]}}]
judge failures
0
grounded weight share
1

Final answer

Now I have the complete picture. The issue is correct - I need to fix my claim about when async scripts execute relative to DOMContentLoaded. Let me provide the corrected answer:

Corrected Answer

How async and defer Attributes Work on HTML Script Elements

The async and defer attributes control when external scripts are fetched and executed, with significant differences in their behavior during HTML parsing, timing relative to the DOMContentLoaded event, and execution order guarantees.

The async Attribute

The async attribute applies to external classic scripts and module scripts. For classic scripts with async, "the classic script will be fetched in parallel to parsing and evaluated as soon as it is available" (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script). This means the browser downloads the script without blocking HTML parsing. However, "once the download is complete, the script will execute, which blocks the page from rendering" (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script). This execution can occur at any time depending on when the download finishes: if the download completes while HTML parsing is still underway, the script executes during parsing; if it completes after parsing finishes, it executes after parsing.

With multiple async scripts, "you can't rely on the order the scripts will load in" (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script). "scripts with an async attribute will execute as soon as the download is complete. This blocks the page and does not guarantee any specific execution order" (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script).

Regarding timing relative to DOMContentLoaded: "The DOMContentLoaded event fires when the HTML document has been completely parsed, and all deferred scripts (<script defer src="…"> and <script type="module">) have downloaded and executed. It doesn't wait for other things like images, subframes, and async scripts to finish loading" (https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event). This means async scripts can execute either before or after DOMContentLoaded fires, depending on whether they finish downloading before or after parsing completes. Importantly, DOMContentLoaded does not wait for async scripts to complete.

The async attribute must not be used on inline scripts: "This attribute must not be used if the src attribute is absent (i.e., for inline scripts) for classic scripts, in this case it would have no effect" (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script).

The defer Attribute

The defer attribute is specified only for external classic scripts. When present, "the classic script will be fetched in parallel and evaluated when the page has finished parsing" (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script). Scripts with defer download in parallel but do not block parsing; they execute after HTML parsing completes. Importantly, "scripts with a defer attribute will load in the order they appear on the page. They won't run until the page content has all loaded, which is useful if your scripts depend on the DOM being in place" (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script). This execution order guarantee is critical when scripts have dependencies on each other.

Regarding DOMContentLoaded timing: "Scripts with the defer attribute will prevent the DOMContentLoaded event from firing until the script has loaded and finished evaluating" (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script). This means deferred scripts execute before DOMContentLoaded fires, and DOMContentLoaded waits for all deferred scripts to complete. The defer attribute cannot be used on inline scripts: "This attribute must not be used if the src attribute is absent (i.e., for inline scripts), in this case it would have no effect" (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script).

Interaction with HTML Parsing

Without async or defer, scripts block HTML parsing entirely. "Scripts without async, defer or type="module" attributes, as well as inline scripts without the type="module" attribute, are fetched and executed immediately before the browser continues to parse the page" (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script). This parser-blocking behavior can significantly delay page rendering. Both async and defer eliminate blocking during the fetch phase: "async and defer allow external scripts to load without blocking the HTML parser while scripts (including inline scripts) with type="module" are deferred automatically" (https://web.dev/learn/performance/optimize-resource-loading). However, async scripts can still block parsing during execution if they finish downloading while parsing is still underway.

The defer attribute does not block parsing during execution; it ensures parsing completes before execution begins.

Inline Scripts vs. External Scripts with src

Inline scripts (those that embed code directly within the <script> tag without a src attribute) cannot use async or defer attributes. "This attribute must not be used if the src attribute is absent (i.e., for inline scripts), in this case it would have no effect" (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script). Inline scripts execute immediately during parsing, blocking further parsing and rendering until they complete. The async and defer attributes only apply to external scripts loaded via the src attribute.

Module Scripts

Module scripts have fundamentally different behavior. "For module scripts, if the async attribute is present, then the module script and all its dependencies will be fetched in parallel to parsing and evaluated as soon as it is available (potentially before parsing completes)" (https://html.spec.whatwg.org/multipage/scripting.html). Without async, "the module script and its dependencies will be fetched in parallel to parsing and evaluated when the page has finished parsing" (https://html.spec.whatwg.org/multipage/scripting.html).

Notably, "The defer attribute has no effect on module scripts — they defer by default" (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script). This means module scripts always load asynchronously (fetched in parallel) and execute after parsing completes, unless the async attribute is explicitly specified. Like deferred scripts, module scripts participate in DOMContentLoaded timing: "The DOMContentLoaded event fires when the HTML document has been completely parsed, and all deferred scripts (<script defer src="…"> and <script type="module">) have downloaded and executed" (https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event).


{"evidence": [{"claim": "async classic scripts are fetched in parallel to parsing and evaluated as soon as they are available", "quote": "the classic script will be fetched in parallel to parsing and evaluated as soon as it is available", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"}, {"claim": "once an async script download completes, the script executes and blocks the page from rendering", "quote": "once the download is complete, the script will execute, which blocks the page from rendering.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"}, {"claim": "async scripts execute as soon as download completes without guaranteeing execution order", "quote": "scripts with an async attribute will execute as soon as the download is complete. This blocks the page and does not guarantee any specific execution order.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"}, {"claim": "you cannot rely on the order async scripts execute in", "quote": "you can't rely on the order the scripts will load in.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"}, {"claim": "DOMContentLoaded does not wait for async scripts to finish loading", "quote": "The DOMContentLoaded event fires when the HTML document has been completely parsed, and all deferred scripts (<script defer src=\"…\"> and <script type=\"module\">) have downloaded and executed. It doesn't wait for other things like images, subframes, and async scripts to finish loading.", "url": "https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event"}, {"claim": "async attribute must not be used on inline scripts (those without src attribute)", "quote": "This attribute must not be used if the src attribute is absent (i.e., for inline scripts) for classic scripts, in this case it would have no effect.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"}, {"claim": "defer classic scripts are fetched in parallel and evaluated when page has finished parsing", "quote": "the classic script will be fetched in parallel and evaluated when the page has finished parsing.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"}, {"claim": "defer scripts execute in the order they appear in the document", "quote": "scripts with a defer attribute will load in the order they appear on the page. They won't run until the page content has all loaded, which is useful if your scripts depend on the DOM being in place", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"}, {"claim": "defer scripts prevent DOMContentLoaded from firing until scripts have loaded and finished evaluating", "quote": "Scripts with the defer attribute will prevent the DOMContentLoaded event from firing until the script has loaded and finished evaluating.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"}, {"claim": "defer attribute must not be used on inline scripts", "quote": "This attribute must not be used if the src attribute is absent (i.e., for inline scripts), in this case it would have no effect.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"}, {"claim": "Scripts without async, defer or type=module are fetched and executed immediately before parsing continues", "quote": "Scripts without async, defer or type=\"module\" attributes, as well as inline scripts without the type=\"module\" attribute, are fetched and executed immediately before the browser continues to parse the page.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"}, {"claim": "async and defer allow external scripts to load without blocking the HTML parser", "quote": "async and defer allow external scripts to load without blocking the HTML parser while scripts (including inline scripts) with type=\"module\" are deferred automatically.", "url": "https://web.dev/learn/performance/optimize-resource-loading"}, {"claim": "Module scripts with async are fetched in parallel and evaluated as soon as available", "quote": "For module scripts, if the async attribute is present, then the module script and all its dependencies will be fetched in parallel to parsing and evaluated as soon as it is available (potentially before parsing completes).", "url": "https://html.spec.whatwg.org/multipage/scripting.html"}, {"claim": "Module scripts without async are fetched in parallel and evaluated when page has finished parsing", "quote": "the module script and its dependencies will be fetched in parallel to parsing and evaluated when the page has finished parsing.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"}, {"claim": "defer attribute has no effect on module scripts as they defer by default", "quote": "The defer attribute has no effect on module scripts — they defer by default.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"}, {"claim": "DOMContentLoaded waits for both defer scripts and module scripts to complete", "quote": "The DOMContentLoaded event fires when the HTML document has been completely parsed, and all deferred scripts (<script defer src=\"…\"> and <script type=\"module\">) have downloaded and executed.", "url": "https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event"}]}

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.