RunSonnet 5.5repeat 2
Plan-execute on async vs defer on <script>
claude-sonnet-5-5__plan-execute__research-script-loading__r2
Score
1.000
- Cost
- $0.169
- + $0.098 judging
- Latency
- 1m 2s
- setup 0.0s, grading 4.5s
- Turns
- 13
- Tool calls
- 10
- Subagents
- 0
- Tokens in / out
- 71k / 8.6k
- 50,429 read from cache
Did the design run?
- Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
- No API failures: heldharness.apiNo model call failed at the API.
- Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
- No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
- Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
- No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
- Builtin tools: heldharness.builtin-toolsA check recorded with the run.
- Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
- No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
- Judge answered: heldharness.judgeAt least one grading call returned a usable verdict.
- The plan was a plan: heldplan-execute.planThe planner’s reply was read through its schema and held at least one step.
- Each call was shown the right tools: heldplan-execute.toolsThe planner was shown no tools and the executor the task’s full toolset.
Phases
Drawn from the phase records: timing and cost per call, no tool calls or messages.
| Phase | Duration | Cost | Turns | Tools | Model |
|---|---|---|---|---|---|
| Plan | 14.0s | $0.021 | 2 | 0 | claude-sonnet-5-5 |
| Execute | 47.5s | $0.148 | 11 | 10 | claude-sonnet-5-5 |
The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.
Runs you can replay
- A gate that blocked the right moveGated-execute · Haiku 4.5
- Three proposals beat one agentPropose-select · Haiku 4.5
- A verifier that earned its keepPlan-execute-verify · Sonnet 5.5
- A revision that did not helpPlan-execute-verify · Haiku 4.5
- The control, missing a stepSingle · Haiku 4.5
Record this cell yourself: npm run bench -- run --strategies plan-execute --tasks research-script-loading --models claude-sonnet-5-5 --repeats 1
Score breakdown
- passStates that both async and defer let a classic script be fetched in parallel with HTML parsing, so the fetch does not block the parser, unlike a classic script with src and neither attribute.1.00weight 0.13
met, backed by verified evidence E1, E2, E3. “the script downloads in parallel with parsing.”
- passStates that a defer script executes only after the document has finished parsing, and before DOMContentLoaded fires.1.00weight 0.19
met, backed by verified evidence E4, E5, E3. “after the document has been parsed, but before firing DOMContentLoaded event”
- passStates that an async script executes as soon as it is available, interrupting parsing if parsing is still in progress, so it may run before or after DOMContentLoaded.1.00weight 0.19
met, backed by verified evidence E2, E17. “an async script may run before or after DOMContentLoaded”
- passStates that multiple defer scripts execute in document order, while async scripts execute in the order their downloads complete, which is not predictable.1.00weight 0.19
met, backed by verified evidence E6, E7, E8. “Scripts with the defer attribute will execute in the order in which they appear in the document.”
- passStates that async and defer have no effect on a classic inline script (a script element without a src attribute).1.00weight 0.13
met, backed by verified evidence E11. “`async` and `defer` do nothing.”
- passStates that module scripts are deferred by default, so the defer attribute has no effect on them, while the async attribute does apply to module scripts.1.00weight 0.19
met, backed by verified evidence E13, E14. “`async` does apply. The module and all its dependencies are fetched in parallel and run as soon as they are ready”
Penalties
No penalties were applied.
Scorer metadata
- corpus
- {"version":2,"snapshot":"2026-10-01","hash":"a5d703ba820ee9c957565ac15b9f98daeac8212372d73d7658a7a6bb2bbc1978","documents":6}
- opened documents
- ["mdn-script-element","whatwg-html-scripting","mdn-domcontentloaded","mdn-js-modules"]
- opened by role
- {"secondary":["mdn-script-element","mdn-domcontentloaded","mdn-js-modules"],"primary":["whatwg-html-scripting"]}
- displayed share
- {"mdn-script-element":0.422,"whatwg-html-scripting":0.2433,"mdn-domcontentloaded":1,"mdn-js-modules":0.0058}
- cited count
- 4
- verified count
- 4
- hallucinated count
- 0
- citation integrity
- 1
- hallucinated urls
- []
- evidence block
- true
- evidence block repaired
- false
- evidence count
- 17
- verified quote count
- 17
- verified quote rate
- 1
- evidence
- [{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Classic script with neither attribute blocks parsing while fetched and evaluated.","quote":"If neither attribute is present, then the script is fetched and evaluated immediately, blocking parsing until these are both complete.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"async classic scripts are fetched in parallel and run as soon as available, possibly before parsing completes.","quote":"For external classic scripts, if the async attribute is present, then the classic script will be fetched in parallel to parsing and evaluated as soon as it is available (potentially before parsing completes).","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"defer classic scripts are fetched in parallel and evaluated after parsing finishes.","quote":"If the async attribute is not present but the defer attribute is present, then the classic script will be fetched in parallel and evaluated when the page has finished parsing.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"Deferred scripts run after parsing but before DOMContentLoaded.","quote":"This Boolean attribute is set to indicate to a browser that the script is meant to be executed after the document has been parsed, but before firing DOMContentLoaded event.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"Deferred scripts delay DOMContentLoaded until they have executed.","quote":"Scripts with the defer attribute will prevent the DOMContentLoaded event from firing until the script has loaded and finished evaluating.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"Deferred scripts preserve document order.","quote":"Scripts with the defer attribute will execute in the order in which they appear in the document.","verified":true,"document":"mdn-script-element"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"The spec keeps async scripts in an unordered set that executes as soon as possible.","quote":"Each Document has a set of scripts that will execute as soon as possible, which is a set of script elements, initially empty.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"The spec keeps deferred (parser-inserted) scripts in a list executed after parsing.","quote":"Each Document has a list of scripts that will execute when the document has finished parsing, which is a list of script elements, initially empty.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event","claim":"DOMContentLoaded does not wait for async scripts.","quote":"It doesn't wait for other things like images, subframes, and async scripts to finish loading.","verified":true,"document":"mdn-domcontentloaded"},{"url":"https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event","claim":"MDN says DOMContentLoaded fires before async scripts execute (stricter than the spec's 'potentially before parsing completes').","quote":"DOMContentLoaded is fired after the execution of these scripts, but before the execution of scripts with async.","verified":true,"document":"mdn-domcontentloaded"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"async and defer have no effect on inline scripts without src.","quote":"Warning: This attribute must not be used if the src attribute is absent (i.e., for inline scripts), in this case it would have no effect.","verified":true,"document":"mdn-script-element"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Inline classic scripts execute immediately.","quote":"Otherwise, immediately execute the script element el, even if other scripts are already executing.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Module scripts are deferred unless async is present, and defer has no effect on them.","quote":"Otherwise, the module script and its dependencies will be fetched in parallel to parsing and evaluated when the page has finished parsing.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules","claim":"Modules are deferred automatically so defer is unnecessary.","quote":"There is no need to use the defer attribute (see <script> attributes) when loading a module script; modules are deferred automatically.","verified":true,"document":"mdn-js-modules"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Inline module scripts are never executed synchronously.","quote":"Queueing a task here means that, even if the inline module script has no dependencies or synchronously results in a parse error, we won't proceed to execute the script element synchronously.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"If both async and defer are given, async wins (MDN); spec says defer is a fallback for legacy browsers.","quote":"If the attribute is specified with the defer attribute, the element will act as if only the async attribute is specified.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event","claim":"A script with async can run after DOMContentLoaded has already fired, per MDN.","quote":"If it's included via <script async>, or it's dynamically injected, then by the time it executes, DOMContentLoaded has already fired.","verified":true,"document":"mdn-domcontentloaded"}]
- judge unverified quotes
- 0
- judge verdicts
- [{"parallel-download":{"status":"met","support":["E1","E2","E3"]},"defer-timing":{"status":"met","support":["E4","E5","E3"]},"async-timing":{"status":"met","support":["E2","E17"]},"ordering":{"status":"met","support":["E6","E7","E8"]},"inline-ignored":{"status":"met","support":["E11"]},"modules":{"status":"met","support":["E13","E14"]}},{"parallel-download":{"status":"met","support":["E1","E2","E3"]},"defer-timing":{"status":"met","support":["E3","E4","E5"]},"async-timing":{"status":"met","support":["E2","E17"]},"ordering":{"status":"met","support":["E6","E7","E8"]},"inline-ignored":{"status":"met","support":["E11"]},"modules":{"status":"met","support":["E13","E14"]}},{"parallel-download":{"status":"met","support":["E1","E2","E3"]},"defer-timing":{"status":"met","support":["E3","E4","E5"]},"async-timing":{"status":"met","support":["E2","E17"]},"ordering":{"status":"met","support":["E6","E7","E8"]},"inline-ignored":{"status":"met","support":["E11"]},"modules":{"status":"met","support":["E13","E14"]}}]
- judge failures
- 0
- grounded weight share
- 1
Final answer
I've read the script-element pages on MDN and in the HTML Standard, plus the MDN pages for DOMContentLoaded and JavaScript modules. I'm writing up the answer now. The javascript.info URL from the plan returned a 404, so I haven't used it.
Answer
Default (neither attribute). A classic <script src> with neither attribute blocks the parser. The spec says "If neither attribute is present, then the script is fetched and evaluated immediately, blocking parsing until these are both complete." (https://html.spec.whatwg.org/multipage/scripting.html). async and defer are both boolean attributes, so the presence of the attribute is what counts (same page).
async.
- Parsing: the script downloads in parallel with parsing. Per the spec, "if the async attribute is present, then the classic script will be fetched in parallel to parsing and evaluated as soon as it is available (potentially before parsing completes)." (https://html.spec.whatwg.org/multipage/scripting.html). Parsing is therefore interrupted only while the script runs, not while it downloads. That follows from the quoted sentence.
- DOMContentLoaded: there is no fixed relationship. MDN says DOMContentLoaded "doesn't wait for other things like images, subframes, and async scripts to finish loading." (https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event).
- Order: the spec keeps async scripts in a set of scripts that will execute as soon as possible, and each one runs when its own result is ready. In the processing algorithm the async branch runs the script when its result is ready and does not check other scripts. So there is no ordering guarantee. This is my reading of the algorithm, not a sentence the spec states.
defer.
- Parsing: the script downloads in parallel and waits. The spec says "If the async attribute is not present but the defer attribute is present, then the classic script will be fetched in parallel and evaluated when the page has finished parsing." (https://html.spec.whatwg.org/multipage/scripting.html).
- DOMContentLoaded: MDN says defer means the script runs "after the document has been parsed, but before firing DOMContentLoaded event". It adds that deferred scripts "will prevent the DOMContentLoaded event from firing until the script has loaded and finished evaluating." (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script). The DOMContentLoaded page agrees: the event "is queued after deferred scripts" (https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event).
- Order: "Scripts with the defer attribute will execute in the order in which they appear in the document." (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script). The spec backs this up by appending each one to a list of scripts that will execute when the document has finished parsing (https://html.spec.whatwg.org/multipage/scripting.html).
Both attributes together. MDN says "If the attribute is specified with the defer attribute, the element will act as if only the async attribute is specified." (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script). The spec says the combination is allowed so that legacy browsers that only support defer fall back to deferred behaviour instead of blocking (https://html.spec.whatwg.org/multipage/scripting.html).
Inline classic scripts (no src). async and defer do nothing. MDN's warning is that the attribute "must not be used if the src attribute is absent (i.e., for inline scripts), in this case it would have no effect." (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script). In the spec, an inline classic script is run at once: "Otherwise, immediately execute the script element el, even if other scripts are already executing." (https://html.spec.whatwg.org/multipage/scripting.html). That branch also covers an external script that has neither attribute and is ready.
type="module" scripts.
- Deferred by default: the spec says that without
async, "the module script and its dependencies will be fetched in parallel to parsing and evaluated when the page has finished parsing." It also says "(The defer attribute has no effect on module scripts.)" (https://html.spec.whatwg.org/multipage/scripting.html). MDN's modules guide agrees: "modules are deferred automatically." (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules). - Order and DOMContentLoaded: modules without
asyncgo into the same list as deferred scripts. They run in document order, before DOMContentLoaded. MDN lists<script type="module">alongsidedeferscripts that DOMContentLoaded waits for. asyncon modules:asyncdoes apply. The module and all its dependencies are fetched in parallel and run as soon as they are ready, potentially before parsing completes (https://html.spec.whatwg.org/multipage/scripting.html). MDN says the same (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script).- Inline modules:
asyncand the module-specific behaviour still apply, because the spec's branch tests "el's type is "module"" and does not requiresrc. An inline module is never run synchronously. The spec notes that "we won't proceed to execute the script element synchronously." So an inline module is deferred, or runs on arrival if it hasasync(https://html.spec.whatwg.org/multipage/scripting.html).
Where the sources differ. The spec says an async script may run "potentially before parsing completes", so it can run before DOMContentLoaded. MDN's DOMContentLoaded page is stricter in two places. One note says the event "is fired after the execution of these scripts, but before the execution of scripts with async." Elsewhere it says that for <script async> "by the time it executes, DOMContentLoaded has already fired." (https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event). I'd treat the spec as authoritative. The practical rule is that an async script may run before or after DOMContentLoaded, so check document.readyState before relying on the event.
Plan departure. The plan's javascript.info source was unavailable, so the DOMContentLoaded claims rest on MDN and the spec.
{"evidence": [
{"claim": "Classic script with neither attribute blocks parsing while fetched and evaluated.", "quote": "If neither attribute is present, then the script is fetched and evaluated immediately, blocking parsing until these are both complete.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "async classic scripts are fetched in parallel and run as soon as available, possibly before parsing completes.", "quote": "For external classic scripts, if the async attribute is present, then the classic script will be fetched in parallel to parsing and evaluated as soon as it is available (potentially before parsing completes).", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "defer classic scripts are fetched in parallel and evaluated after parsing finishes.", "quote": "If the async attribute is not present but the defer attribute is present, then the classic script will be fetched in parallel and evaluated when the page has finished parsing.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Deferred scripts run after parsing but before DOMContentLoaded.", "quote": "This Boolean attribute is set to indicate to a browser that the script is meant to be executed after the document has been parsed, but before firing DOMContentLoaded event.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "Deferred scripts delay DOMContentLoaded until they have executed.", "quote": "Scripts with the defer attribute will prevent the DOMContentLoaded event from firing until the script has loaded and finished evaluating.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "Deferred scripts preserve document order.", "quote": "Scripts with the defer attribute will execute in the order in which they appear in the document.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "The spec keeps async scripts in an unordered set that executes as soon as possible.", "quote": "Each Document has a set of scripts that will execute as soon as possible, which is a set of script elements, initially empty.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "The spec keeps deferred (parser-inserted) scripts in a list executed after parsing.", "quote": "Each Document has a list of scripts that will execute when the document has finished parsing, which is a list of script elements, initially empty.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "DOMContentLoaded does not wait for async scripts.", "quote": "It doesn't wait for other things like images, subframes, and async scripts to finish loading.", "url": "https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event"},
{"claim": "MDN says DOMContentLoaded fires before async scripts execute (stricter than the spec's 'potentially before parsing completes').", "quote": "DOMContentLoaded is fired after the execution of these scripts, but before the execution of scripts with async.", "url": "https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event"},
{"claim": "async and defer have no effect on inline scripts without src.", "quote": "Warning: This attribute must not be used if the src attribute is absent (i.e., for inline scripts), in this case it would have no effect.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "Inline classic scripts execute immediately.", "quote": "Otherwise, immediately execute the script element el, even if other scripts are already executing.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Module scripts are deferred unless async is present, and defer has no effect on them.", "quote": "Otherwise, the module script and its dependencies will be fetched in parallel to parsing and evaluated when the page has finished parsing.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Modules are deferred automatically so defer is unnecessary.", "quote": "There is no need to use the defer attribute (see <script> attributes) when loading a module script; modules are deferred automatically.", "url": "https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules"},
{"claim": "Inline module scripts are never executed synchronously.", "quote": "Queueing a task here means that, even if the inline module script has no dependencies or synchronously results in a parse error, we won't proceed to execute the script element synchronously.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "If both async and defer are given, async wins (MDN); spec says defer is a fallback for legacy browsers.", "quote": "If the attribute is specified with the defer attribute, the element will act as if only the async attribute is specified.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "A script with async can run after DOMContentLoaded has already fired, per MDN.", "quote": "If it's included via <script async>, or it's dynamically injected, then by the time it executes, DOMContentLoaded has already fired.", "url": "https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event"}
]}Provenance of the numbers on this page
- run set
- scaffold-or-scale-v1
- models
- Haiku 4.5, Sonnet 5.5, Opus 5.5
- effort
- high
- repeats
- 2
- fixtures
- replay
- runs
- 900 / 900
- strategy cost
- $354.44
- judge cost
- $9.19
- wall clock
- 9h 48m
- git
- fce5807b2d
Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.