RunSonnet 5.5repeat 1
Plan-execute on async vs defer on <script>
claude-sonnet-5-5__plan-execute__research-script-loading__r1
Score
1.000
- Cost
- $0.223
- + $0.078 judging
- Latency
- 56.1s
- setup 0.0s, grading 4.9s
- Turns
- 22
- Tool calls
- 19
- Subagents
- 0
- Tokens in / out
- 189k / 7.6k
- 157,627 read from cache
Did the design run?
- Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
- No API failures: heldharness.apiNo model call failed at the API.
- Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
- No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
- Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
- No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
- Builtin tools: heldharness.builtin-toolsA check recorded with the run.
- Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
- No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
- Judge answered: heldharness.judgeAt least one grading call returned a usable verdict.
- The plan was a plan: heldplan-execute.planThe planner’s reply was read through its schema and held at least one step.
- Each call was shown the right tools: heldplan-execute.toolsThe planner was shown no tools and the executor the task’s full toolset.
Phases
Drawn from the phase records: timing and cost per call, no tool calls or messages.
| Phase | Duration | Cost | Turns | Tools | Model |
|---|---|---|---|---|---|
| Plan | 12.8s | $0.019 | 2 | 0 | claude-sonnet-5-5 |
| Execute | 43.4s | $0.204 | 20 | 19 | claude-sonnet-5-5 |
The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.
Runs you can replay
- A gate that blocked the right moveGated-execute · Haiku 4.5
- Three proposals beat one agentPropose-select · Haiku 4.5
- A verifier that earned its keepPlan-execute-verify · Sonnet 5.5
- A revision that did not helpPlan-execute-verify · Haiku 4.5
- The control, missing a stepSingle · Haiku 4.5
Record this cell yourself: npm run bench -- run --strategies plan-execute --tasks research-script-loading --models claude-sonnet-5-5 --repeats 1
Score breakdown
- passStates that both async and defer let a classic script be fetched in parallel with HTML parsing, so the fetch does not block the parser, unlike a classic script with src and neither attribute.1.00weight 0.13
met, backed by verified evidence E1, E2, E3. “`async` and `defer` both let an external script download while the HTML parser keeps going.”
- passStates that a defer script executes only after the document has finished parsing, and before DOMContentLoaded fires.1.00weight 0.19
met, backed by verified evidence E3, E5, E8. “after the document has been parsed, but before firing DOMContentLoaded event”
- passStates that an async script executes as soon as it is available, interrupting parsing if parsing is still in progress, so it may run before or after DOMContentLoaded.1.00weight 0.19
met, backed by verified evidence E2, E7, E15. “evaluated as soon as it is available (potentially before parsing completes)”
- passStates that multiple defer scripts execute in document order, while async scripts execute in the order their downloads complete, which is not predictable.1.00weight 0.19
met, backed by verified evidence E6, E14. “Deferred scripts run in document order.”
- passStates that async and defer have no effect on a classic inline script (a script element without a src attribute).1.00weight 0.13
met, backed by verified evidence E9. “`async` and `defer` do nothing.”
- passStates that module scripts are deferred by default, so the defer attribute has no effect on them, while the async attribute does apply to module scripts.1.00weight 0.19
met, backed by verified evidence E4, E12. “The defer attribute has no effect on module scripts”
Penalties
No penalties were applied.
Scorer metadata
- corpus
- {"version":2,"snapshot":"2026-10-01","hash":"a5d703ba820ee9c957565ac15b9f98daeac8212372d73d7658a7a6bb2bbc1978","documents":6}
- opened documents
- ["mdn-script-element","mdn-domcontentloaded","whatwg-html-scripting"]
- opened by role
- {"secondary":["mdn-script-element","mdn-domcontentloaded"],"primary":["whatwg-html-scripting"]}
- displayed share
- {"mdn-script-element":0.6195,"mdn-domcontentloaded":0.9656,"whatwg-html-scripting":0.3338,"webdev-third-party-javascript":0.0309,"webdev-optimize-resource-loading":0.0155}
- cited count
- 3
- verified count
- 3
- hallucinated count
- 0
- citation integrity
- 1
- hallucinated urls
- []
- evidence block
- true
- evidence block repaired
- false
- evidence count
- 15
- verified quote count
- 15
- verified quote rate
- 1
- evidence
- [{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"A classic external script with neither async nor defer is fetched and run immediately, blocking the parser.","quote":"If neither attribute is present, then the script is fetched and evaluated immediately, blocking parsing until these are both complete.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Classic async scripts are fetched in parallel with parsing and run as soon as available, possibly before parsing completes.","quote":"For external classic scripts, if the async attribute is present, then the classic script will be fetched in parallel to parsing and evaluated as soon as it is available (potentially before parsing completes).","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Classic defer scripts are fetched in parallel and evaluated once parsing has finished.","quote":"If the async attribute is not present but the defer attribute is present, then the classic script will be fetched in parallel and evaluated when the page has finished parsing.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Module scripts are deferred by default, with async making them run as soon as available; defer has no effect on them.","quote":"Otherwise, the module script and its dependencies will be fetched in parallel to parsing and evaluated when the page has finished parsing. (The defer attribute has no effect on module scripts.)","verified":true,"document":"whatwg-html-scripting"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"Deferred scripts run before DOMContentLoaded and hold that event back until they finish.","quote":"Scripts with the defer attribute will prevent the DOMContentLoaded event from firing until the script has loaded and finished evaluating.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"Deferred scripts execute in document order.","quote":"Scripts with the defer attribute will execute in the order in which they appear in the document.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event","claim":"DOMContentLoaded waits for defer and module scripts but not async scripts.","quote":"It doesn't wait for other things like images, subframes, and async scripts to finish loading.","verified":true,"document":"mdn-domcontentloaded"},{"url":"https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event","claim":"DOMContentLoaded fires after defer/module scripts but before async scripts; readyState becomes complete after async scripts.","quote":"DOMContentLoaded is fired after the execution of these scripts, but before the execution of scripts with async.","verified":true,"document":"mdn-domcontentloaded"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"async and defer have no effect on classic inline scripts.","quote":"This attribute must not be used if the src attribute is absent (i.e., for inline scripts), in this case it would have no effect.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"If both async and defer are present, the element acts as async only.","quote":"If the attribute is specified with the defer attribute, the element will act as if only the async attribute is specified.","verified":true,"document":"mdn-script-element"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Setting defer alongside async exists as a fallback for legacy browsers that only support defer.","quote":"The defer attribute may be specified even if the async attribute is specified, to cause legacy web browsers that only support defer (and not async) to fall back to the defer behavior instead of the blocking behavior that is the default.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"Module scripts with async are fetched along with all dependencies in parallel and run as soon as available.","quote":"For module scripts, if the async attribute is present then the scripts and all their dependencies will be fetched in parallel to parsing and evaluated as soon as they are available.","verified":true,"document":"mdn-script-element"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Dynamically inserted (non-parser-inserted) scripts have force async true by default, so they behave as async.","quote":"A script element has a force async boolean, initially true.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Async scripts are tracked in an unordered set, and parser-inserted deferred scripts in a list that runs after parsing.","quote":"Each Document has a set of scripts that will execute as soon as possible, which is a set of script elements, initially empty.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event","claim":"Async scripts can run after DOMContentLoaded has already fired.","quote":"If it's included via <script async>, or it's dynamically injected, then by the time it executes, DOMContentLoaded has already fired.","verified":true,"document":"mdn-domcontentloaded"}]
- judge unverified quotes
- 0
- judge verdicts
- [{"parallel-download":{"status":"met","support":["E1","E2","E3"]},"defer-timing":{"status":"met","support":["E3","E5","E8"]},"async-timing":{"status":"met","support":["E2","E7","E15"]},"ordering":{"status":"met","support":["E6","E14"]},"inline-ignored":{"status":"met","support":["E9"]},"modules":{"status":"met","support":["E4","E12"]}},{"parallel-download":{"status":"met","support":["E1","E2","E3"]},"defer-timing":{"status":"met","support":["E3","E5","E8"]},"async-timing":{"status":"met","support":["E2","E7","E15"]},"ordering":{"status":"met","support":["E6","E14"]},"inline-ignored":{"status":"met","support":["E9"]},"modules":{"status":"met","support":["E4","E12"]}},{"parallel-download":{"status":"met","support":["E1","E2","E3"]},"defer-timing":{"status":"met","support":["E3","E5","E8"]},"async-timing":{"status":"met","support":["E2","E7","E15"]},"ordering":{"status":"met","support":["E6","E14"]},"inline-ignored":{"status":"met","support":["E9"]},"modules":{"status":"met","support":["E4","E12"]}}]
- judge failures
- 0
- grounded weight share
- 1
Final answer
async and defer both let an external script download while the HTML parser keeps going. They differ in when the script runs: async runs as soon as the file arrives, and defer waits until parsing has finished. I read MDN and the HTML Standard's scripting section. I could not open the Standard's parsing page, because every URL I tried there returned a 404. So the DOMContentLoaded timing comes from MDN only.
Plain classic <script src> (no attributes). The script is fetched and run immediately, and the parser waits for it. The Standard says "If neither attribute is present, then the script is fetched and evaluated immediately, blocking parsing until these are both complete." (https://html.spec.whatwg.org/multipage/scripting.html) MDN agrees (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script).
defer (external classic scripts).
- Parsing: The script is fetched in parallel with parsing and evaluated "when the page has finished parsing" (https://html.spec.whatwg.org/multipage/scripting.html).
- DOMContentLoaded: MDN says defer scripts run "after the document has been parsed, but before firing DOMContentLoaded event" and that they hold that event back until they have loaded and finished running (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script). The DOMContentLoaded page says the event fires after the deferred scripts and module scripts have executed (https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event).
- Order: Deferred scripts run in document order. MDN states this directly. The Standard's processing model matches: parser-inserted deferred scripts are appended to a "list of scripts that will execute when the document has finished parsing", and a list is ordered (https://html.spec.whatwg.org/multipage/scripting.html).
async (external classic scripts).
- Parsing: The script is fetched in parallel with parsing and "evaluated as soon as it is available (potentially before parsing completes)". Parsing pauses only while the script runs.
- Order: There is no ordering guarantee. The Standard puts async scripts into a set ("set of scripts that will execute as soon as possible") and runs each one when its own result is ready. Whichever file arrives first runs first. This is my reading of the processing model, not a sentence the spec states outright.
- DOMContentLoaded: There is no fixed relationship. MDN says DOMContentLoaded doesn't wait for "async scripts to finish loading". An async script may therefore run before the event or after it, and MDN notes it can run after
DOMContentLoadedhas already fired. MDN saysdocument.readyStatebecomes "complete" after async scripts execute.
Both attributes set.
- MDN says
deferplusasyncmakes the element "act as if only the async attribute is specified". - The Standard explains the reason. Authors may set both so that old browsers that only know
deferfall back to deferred behavior instead of blocking. - The two statements are consistent. I read them as the same rule from two angles, not a conflict.
Inline scripts.
- Classic inline scripts:
asyncanddeferdo nothing. MDN's wording is that the attribute "must not be used if the src attribute is absent… in this case it would have no effect". The Standard's algorithm agrees: its async, in-order and deferred branches apply only to classic scripts withsrc. A classic inline script otherwise executes immediately (https://html.spec.whatwg.org/multipage/scripting.html). - Inline module scripts: These behave differently, as described in the next section.
type="module".
- Default timing: Module scripts defer by default. The Standard says the module and its dependencies are fetched in parallel and "evaluated when the page has finished parsing", and "The defer attribute has no effect on module scripts". MDN says the same.
- With
async: The module and all its dependencies are fetched in parallel and the module runs as soon as it is available, possibly before parsing completes. This is the same behavior as async classic scripts. - DOMContentLoaded: MDN lists
<script type="module">among the deferred scripts that DOMContentLoaded waits for. - Inline modules: The Standard's algorithm treats a module script whether or not it has
src. It deferred-queues inline modules, andasyncapplies to them too. Theasyncwarning on MDN is limited to classic scripts. The Standard also queues a task even for an inline module, so it never runs synchronously.
Dynamically inserted scripts. A script created by document.createElement has a "force async" flag that starts out true. So it acts as async unless you set script.async = false. The Standard's algorithm then runs non-async, non-parser-inserted external scripts in insertion order, via the "list of scripts that will execute in order as soon as possible". MDN adds that dynamically injected scripts can run after DOMContentLoaded has already fired.
Where the sources differ or are thin. The main gap is that I did not read the Standard's parsing page, so the DOMContentLoaded relationship rests on MDN. MDN also words the inline-script case as an async and defer "warning", while the Standard encodes it only through the algorithm. Neither source contradicts the other on substance.
{"evidence": [
{"claim": "A classic external script with neither async nor defer is fetched and run immediately, blocking the parser.", "quote": "If neither attribute is present, then the script is fetched and evaluated immediately, blocking parsing until these are both complete.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Classic async scripts are fetched in parallel with parsing and run as soon as available, possibly before parsing completes.", "quote": "For external classic scripts, if the async attribute is present, then the classic script will be fetched in parallel to parsing and evaluated as soon as it is available (potentially before parsing completes).", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Classic defer scripts are fetched in parallel and evaluated once parsing has finished.", "quote": "If the async attribute is not present but the defer attribute is present, then the classic script will be fetched in parallel and evaluated when the page has finished parsing.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Module scripts are deferred by default, with async making them run as soon as available; defer has no effect on them.", "quote": "Otherwise, the module script and its dependencies will be fetched in parallel to parsing and evaluated when the page has finished parsing. (The defer attribute has no effect on module scripts.)", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Deferred scripts run before DOMContentLoaded and hold that event back until they finish.", "quote": "Scripts with the defer attribute will prevent the DOMContentLoaded event from firing until the script has loaded and finished evaluating.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "Deferred scripts execute in document order.", "quote": "Scripts with the defer attribute will execute in the order in which they appear in the document.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "DOMContentLoaded waits for defer and module scripts but not async scripts.", "quote": "It doesn't wait for other things like images, subframes, and async scripts to finish loading.", "url": "https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event"},
{"claim": "DOMContentLoaded fires after defer/module scripts but before async scripts; readyState becomes complete after async scripts.", "quote": "DOMContentLoaded is fired after the execution of these scripts, but before the execution of scripts with async.", "url": "https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event"},
{"claim": "async and defer have no effect on classic inline scripts.", "quote": "This attribute must not be used if the src attribute is absent (i.e., for inline scripts), in this case it would have no effect.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "If both async and defer are present, the element acts as async only.", "quote": "If the attribute is specified with the defer attribute, the element will act as if only the async attribute is specified.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "Setting defer alongside async exists as a fallback for legacy browsers that only support defer.", "quote": "The defer attribute may be specified even if the async attribute is specified, to cause legacy web browsers that only support defer (and not async) to fall back to the defer behavior instead of the blocking behavior that is the default.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Module scripts with async are fetched along with all dependencies in parallel and run as soon as available.", "quote": "For module scripts, if the async attribute is present then the scripts and all their dependencies will be fetched in parallel to parsing and evaluated as soon as they are available.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "Dynamically inserted (non-parser-inserted) scripts have force async true by default, so they behave as async.", "quote": "A script element has a force async boolean, initially true.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Async scripts are tracked in an unordered set, and parser-inserted deferred scripts in a list that runs after parsing.", "quote": "Each Document has a set of scripts that will execute as soon as possible, which is a set of script elements, initially empty.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Async scripts can run after DOMContentLoaded has already fired.", "quote": "If it's included via <script async>, or it's dynamically injected, then by the time it executes, DOMContentLoaded has already fired.", "url": "https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event"}
]}Provenance of the numbers on this page
- run set
- scaffold-or-scale-v1
- models
- Haiku 4.5, Sonnet 5.5, Opus 5.5
- effort
- high
- repeats
- 2
- fixtures
- replay
- runs
- 900 / 900
- strategy cost
- $354.44
- judge cost
- $9.19
- wall clock
- 9h 48m
- git
- fce5807b2d
Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.