RunHaiku 4.5repeat 2
Plan-execute-verify on async vs defer on <script>
claude-haiku-4-5-20251001__plan-execute-verify__research-script-loading__r2
Score
0.906
- Cost
- $0.259
- + $0.076 judging
- Latency
- 3m 19s
- setup 0.0s, grading 4.6s
- Turns
- 34
- Tool calls
- 28
- Subagents
- 0
- Tokens in / out
- 314k / 20k
- 238,773 read from cache
Did the design run?
- Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
- No API failures: heldharness.apiNo model call failed at the API.
- Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
- No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
- Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
- No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
- Builtin tools: heldharness.builtin-toolsA check recorded with the run.
- Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
- No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
- Judge answered: heldharness.judgeAt least one grading call returned a usable verdict.
- The plan was a plan: heldplan-execute-verify.planThe planner’s reply was read through its schema and held at least one step.
- Every verdict was read: heldplan-execute-verify.verdictsEvery review round ended with a verdict read from the schema reply, not a default.
- The loop did what its verdicts said: heldplan-execute-verify.loopEvery verdict that called for a revision got one, and nothing was revised without such a verdict.
- Each call was shown the right tools: heldplan-execute-verify.toolsThe planner was shown no tools, the verifier the read-only ones, and the executor and reviser the full toolset.
Phases
Drawn from the phase records: timing and cost per call, no tool calls or messages.
| Phase | Duration | Cost | Turns | Tools | Model |
|---|---|---|---|---|---|
| Plan | 17.4s | $0.015 | 3 | 0 | claude-haiku-4-5-20251001 |
| Execute | 54.7s | $0.092 | 15 | 14 | claude-haiku-4-5-20251001 |
| Verify | 2m 7s | $0.152 | 16 | 14 | claude-haiku-4-5-20251001 |
The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.
Runs you can replay
- A gate that blocked the right moveGated-execute · Haiku 4.5
- Three proposals beat one agentPropose-select · Haiku 4.5
- A verifier that earned its keepPlan-execute-verify · Sonnet 5.5
- A revision that did not helpPlan-execute-verify · Haiku 4.5
- The control, missing a stepSingle · Haiku 4.5
Record this cell yourself: npm run bench -- run --strategies plan-execute-verify --tasks research-script-loading --models claude-haiku-4-5-20251001 --repeats 1
Score breakdown
- passStates that both async and defer let a classic script be fetched in parallel with HTML parsing, so the fetch does not block the parser, unlike a classic script with src and neither attribute.1.00weight 0.13
met, backed by verified evidence E1, E4. “`defer` scripts are also fetched in parallel but defer all execution until parsing is complete, never blocking the parser. Normal scripts without either attribute "block parsing until these are both complete"”
- passStates that a defer script executes only after the document has finished parsing, and before DOMContentLoaded fires.1.00weight 0.19
met, backed by verified evidence E4, E7. “This means `defer` scripts execute before DOMContentLoaded fires”
- partialStates that an async script executes as soon as it is available, interrupting parsing if parsing is still in progress, so it may run before or after DOMContentLoaded.0.50weight 0.19
partial, backed by verified evidence E1, E8. “"DOMContentLoaded is fired after the execution of these scripts, but before the execution of scripts with async"”
- passStates that multiple defer scripts execute in document order, while async scripts execute in the order their downloads complete, which is not predictable.1.00weight 0.19
met, backed by verified evidence E2, E5. “scripts execute in the order they finish downloading, not the order they appear in the HTML”
- passStates that async and defer have no effect on a classic inline script (a script element without a src attribute).1.00weight 0.13
met, backed by verified evidence E3, E6. “`async` and `defer` attributes have no effect on them.”
- passStates that module scripts are deferred by default, so the defer attribute has no effect on them, while the async attribute does apply to module scripts.1.00weight 0.19
met, backed by verified evidence E9, E10. “module scripts defer by default—they behave like `defer` scripts without the `async` attribute. With `async`, they behave like async scripts.”
Penalties
No penalties were applied.
Scorer metadata
- corpus
- {"version":2,"snapshot":"2026-10-01","hash":"a5d703ba820ee9c957565ac15b9f98daeac8212372d73d7658a7a6bb2bbc1978","documents":6}
- opened documents
- ["mdn-script-element","mdn-domcontentloaded","webdev-optimize-resource-loading","whatwg-html-scripting","webdev-third-party-javascript"]
- opened by role
- {"secondary":["mdn-script-element","mdn-domcontentloaded","webdev-optimize-resource-loading","webdev-third-party-javascript"],"primary":["whatwg-html-scripting"]}
- displayed share
- {"whatwg-html-scripting":0.2434,"webdev-third-party-javascript":0.9618,"mdn-script-element":1,"webdev-optimize-resource-loading":1,"mdn-domcontentloaded":1}
- cited count
- 4
- verified count
- 4
- hallucinated count
- 0
- citation integrity
- 1
- hallucinated urls
- []
- evidence block
- true
- evidence block repaired
- false
- evidence count
- 12
- verified quote count
- 12
- verified quote rate
- 1
- evidence
- [{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"For external classic scripts, async causes the script to be fetched in parallel to parsing and evaluated as soon as it is available","quote":"For external classic scripts, if the async attribute is present, then the classic script will be fetched in parallel to parsing and evaluated as soon as it is available (potentially before parsing completes).","verified":true,"document":"whatwg-html-scripting"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"Async scripts execute as soon as download is complete and block the page, with no guaranteed execution order for multiple async scripts","quote":"scripts with an async attribute will execute as soon as the download is complete. This blocks the page and does not guarantee any specific execution order.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"Async must not be used on inline scripts because it has no effect","quote":"This attribute must not be used if the src attribute is absent (i.e., for inline scripts) for classic scripts, in this case it would have no effect.","verified":true,"document":"mdn-script-element"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Defer causes external classic scripts to be fetched in parallel and evaluated when the page has finished parsing","quote":"If the async attribute is not present but the defer attribute is present, then the classic script will be fetched in parallel and evaluated when the page has finished parsing.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"Scripts with defer attribute execute in the order they appear in the document","quote":"Scripts with the defer attribute will execute in the order in which they appear in the document.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"Defer must not be used on inline scripts because it has no effect","quote":"This attribute must not be used if the src attribute is absent (i.e., for inline scripts), in this case it would have no effect.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event","claim":"DOMContentLoaded fires after deferred scripts and module scripts have executed, but before async scripts","quote":"The DOMContentLoaded event fires when the HTML document has been completely parsed, and all deferred scripts (<script defer src=\"…\"> and <script type=\"module\">) have downloaded and executed.","verified":true,"document":"mdn-domcontentloaded"},{"url":"https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event","claim":"DOMContentLoaded fires after deferred and module script execution, but before async script execution","quote":"DOMContentLoaded is fired after the execution of these scripts, but before the execution of scripts with async.","verified":true,"document":"mdn-domcontentloaded"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Module scripts defer by default; async attribute can override this default deferring behavior","quote":"For module scripts, if the async attribute is present, then the module script and all its dependencies will be fetched in parallel to parsing, and the module script will be evaluated as soon as it is available (potentially before parsing completes). Otherwise, the module script and its dependencies \n…[truncated 143 chars]","verified":true,"document":"whatwg-html-scripting"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"Defer attribute has no effect on module scripts because they defer by default","quote":"The defer attribute has no effect on module scripts — they defer by default.","verified":true,"document":"mdn-script-element"},{"url":"https://web.dev/learn/performance/optimize-resource-loading","claim":"Inline module scripts defer by default like external module scripts","quote":"scripts (including inline scripts) with type=\"module\" are deferred automatically.","verified":true,"document":"webdev-optimize-resource-loading"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"When both async and defer are specified, async takes precedence","quote":"If the attribute is specified with the async attribute, the element will act as if only the async attribute is specified.","verified":true,"document":"mdn-script-element"}]
- judge unverified quotes
- 0
- judge verdicts
- [{"parallel-download":{"status":"met","support":["E1","E4"]},"defer-timing":{"status":"met","support":["E4","E7"]},"async-timing":{"status":"partial","support":["E1","E8"]},"ordering":{"status":"met","support":["E2","E5"]},"inline-ignored":{"status":"met","support":["E3","E6"]},"modules":{"status":"met","support":["E9","E10"]}},{"parallel-download":{"status":"met","support":["E1","E4"]},"defer-timing":{"status":"met","support":["E4","E7"]},"async-timing":{"status":"partial","support":["E1","E8"]},"ordering":{"status":"met","support":["E2","E5"]},"inline-ignored":{"status":"met","support":["E3","E6"]},"modules":{"status":"met","support":["E9","E10"]}},{"parallel-download":{"status":"met","support":["E1","E4"]},"defer-timing":{"status":"met","support":["E4","E7"]},"async-timing":{"status":"partial","support":["E1","E8"]},"ordering":{"status":"met","support":["E2","E5"]},"inline-ignored":{"status":"met","support":["E3","E6"]},"modules":{"status":"met","support":["E9","E10"]}}]
- judge failures
- 0
- grounded weight share
- 1
Final answer
Based on my research, I now have comprehensive information about the async and defer attributes. Let me compile a complete answer.
Complete Answer
The async and defer attributes on HTML <script> elements control when scripts are fetched and executed relative to HTML parsing and the DOMContentLoaded event. They have fundamentally different behaviors despite both allowing non-blocking fetches.
The async Attribute
For external classic scripts, the async attribute causes the script to be "fetched in parallel to parsing and evaluated as soon as it is available (potentially before parsing completes)" (https://html.spec.whatwg.org/multipage/scripting.html). This means the browser downloads the script without blocking HTML parsing, but executes it immediately upon completion, which can interrupt DOM construction. Importantly, async provides no guarantee about execution order when multiple scripts use this attribute; scripts execute in the order they finish downloading, not the order they appear in the HTML. Additionally, "scripts with an async attribute will execute as soon as the download is complete. This blocks the page and does not guarantee any specific execution order" (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script).
The async attribute "must not be used if the src attribute is absent (i.e., for inline scripts) for classic scripts, in this case it would have no effect" (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script).
The defer Attribute
For external classic scripts, defer causes the script to be "fetched in parallel and evaluated when the page has finished parsing" (https://html.spec.whatwg.org/multipage/scripting.html). This means "scripts with a defer attribute will load in the order they are in and will only execute once everything has finished loading" (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script). Critically, "Scripts with the defer attribute will execute in the order in which they appear in the document" (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script), providing guaranteed execution order.
Like async, "This attribute must not be used if the src attribute is absent (i.e., for inline scripts), in this case it would have no effect" (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script).
Interaction with HTML Parsing
The critical difference in parsing interaction is straightforward: async scripts are fetched in parallel but execute immediately when ready, potentially blocking the parser. defer scripts are also fetched in parallel but defer all execution until parsing is complete, never blocking the parser. Normal scripts without either attribute "block parsing until these are both complete" (https://html.spec.whatwg.org/multipage/scripting.html).
DOMContentLoaded Event Timing
The DOMContentLoaded event fires at different points depending on script attributes. "The DOMContentLoaded event fires when the HTML document has been completely parsed, and all deferred scripts (<script defer src="…"> and <script type="module">) have downloaded and executed" (https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event). This means defer scripts execute before DOMContentLoaded fires, but "DOMContentLoaded is fired after the execution of these scripts, but before the execution of scripts with async" (https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event).
Inline Scripts
Inline scripts (scripts with inline content rather than a src attribute) are fundamentally different: async and defer attributes have no effect on them. Inline scripts with no type attribute execute immediately during HTML parsing, blocking the parser. However, "inline scripts with the type="module" attribute" do defer by default (https://web.dev/learn/performance/optimize-resource-loading).
Module Scripts (type="module")
Module scripts have special behavior. For module scripts, "if the async attribute is present, then the module script and all its dependencies will be fetched in parallel to parsing, and the module script will be evaluated as soon as it is available (potentially before parsing completes). Otherwise, the module script and its dependencies will be fetched in parallel to parsing and evaluated when the page has finished parsing. (The defer attribute has no effect on module scripts.)" (https://html.spec.whatwg.org/multipage/scripting.html).
This means module scripts defer by default—they behave like defer scripts without the async attribute. With async, they behave like async scripts. The key difference from classic scripts is that "The defer attribute has no effect on module scripts — they defer by default" (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script).
When Both Attributes Are Specified
If both async and defer are specified, the async behavior takes precedence. "If the attribute is specified with the async attribute, the element will act as if only the async attribute is specified" (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script).
{"evidence": [{"claim": "For external classic scripts, async causes the script to be fetched in parallel to parsing and evaluated as soon as it is available", "quote": "For external classic scripts, if the async attribute is present, then the classic script will be fetched in parallel to parsing and evaluated as soon as it is available (potentially before parsing completes).", "url": "https://html.spec.whatwg.org/multipage/scripting.html"}, {"claim": "Async scripts execute as soon as download is complete and block the page, with no guaranteed execution order for multiple async scripts", "quote": "scripts with an async attribute will execute as soon as the download is complete. This blocks the page and does not guarantee any specific execution order.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"}, {"claim": "Async must not be used on inline scripts because it has no effect", "quote": "This attribute must not be used if the src attribute is absent (i.e., for inline scripts) for classic scripts, in this case it would have no effect.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"}, {"claim": "Defer causes external classic scripts to be fetched in parallel and evaluated when the page has finished parsing", "quote": "If the async attribute is not present but the defer attribute is present, then the classic script will be fetched in parallel and evaluated when the page has finished parsing.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"}, {"claim": "Scripts with defer attribute execute in the order they appear in the document", "quote": "Scripts with the defer attribute will execute in the order in which they appear in the document.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"}, {"claim": "Defer must not be used on inline scripts because it has no effect", "quote": "This attribute must not be used if the src attribute is absent (i.e., for inline scripts), in this case it would have no effect.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"}, {"claim": "DOMContentLoaded fires after deferred scripts and module scripts have executed, but before async scripts", "quote": "The DOMContentLoaded event fires when the HTML document has been completely parsed, and all deferred scripts (<script defer src=\"…\"> and <script type=\"module\">) have downloaded and executed.", "url": "https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event"}, {"claim": "DOMContentLoaded fires after deferred and module script execution, but before async script execution", "quote": "DOMContentLoaded is fired after the execution of these scripts, but before the execution of scripts with async.", "url": "https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event"}, {"claim": "Module scripts defer by default; async attribute can override this default deferring behavior", "quote": "For module scripts, if the async attribute is present, then the module script and all its dependencies will be fetched in parallel to parsing, and the module script will be evaluated as soon as it is available (potentially before parsing completes). Otherwise, the module script and its dependencies will be fetched in parallel to parsing and evaluated when the page has finished parsing. (The defer attribute has no effect on module scripts.)", "url": "https://html.spec.whatwg.org/multipage/scripting.html"}, {"claim": "Defer attribute has no effect on module scripts because they defer by default", "quote": "The defer attribute has no effect on module scripts — they defer by default.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"}, {"claim": "Inline module scripts defer by default like external module scripts", "quote": "scripts (including inline scripts) with type=\"module\" are deferred automatically.", "url": "https://web.dev/learn/performance/optimize-resource-loading"}, {"claim": "When both async and defer are specified, async takes precedence", "quote": "If the attribute is specified with the async attribute, the element will act as if only the async attribute is specified.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"}]}Provenance of the numbers on this page
- run set
- scaffold-or-scale-v1
- models
- Haiku 4.5, Sonnet 5.5, Opus 5.5
- effort
- high
- repeats
- 2
- fixtures
- replay
- runs
- 900 / 900
- strategy cost
- $354.44
- judge cost
- $9.19
- wall clock
- 9h 48m
- git
- fce5807b2d
Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.