RunSonnet 5.5repeat 2
Propose-select on async vs defer on <script>
claude-sonnet-5-5__propose-select__research-script-loading__r2
Score
0.938
- Cost
- $0.570
- + $0.101 judging
- Latency
- 1m 14s
- setup 0.0s, grading 5.2s
- Turns
- 45
- Tool calls
- 40
- Subagents
- 3
- Tokens in / out
- 330k / 23k
- 247,969 read from cache
Did the design run?
- Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
- No API failures: heldharness.apiNo model call failed at the API.
- Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
- No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
- Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
- No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
- Builtin tools: heldharness.builtin-toolsA check recorded with the run.
- Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
- No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
- Judge answered: heldharness.judgeAt least one grading call returned a usable verdict.
- All three proposers ran: heldpropose-select.proposalsThree proposers ran in parallel, each from its own angle, and none was starved while another ran.
- The selector chose: heldpropose-select.selectionThe selector picked a proposal by a valid number, read from its schema reply and not a default.
- Each call was shown the right tools: heldpropose-select.toolsEach proposer was shown the read-only tools, the selector none, and the executor the full toolset.
Phases
Drawn from the phase records: timing and cost per call, no tool calls or messages.
| Phase | Duration | Cost | Turns | Tools | Model |
|---|---|---|---|---|---|
| Proposal 1 (policy-first) | 42.9s | $0.151 | 15 | 14 | claude-sonnet-5-5 |
| Proposal 2 (state-first) | 42.0s | $0.168 | 14 | 13 | claude-sonnet-5-5 |
| Proposal 3 (outcome-first) | 39.7s | $0.137 | 14 | 13 | claude-sonnet-5-5 |
| Select | 31.1s | $0.114 | 2 | 0 | claude-sonnet-5-5 |
The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.
Runs you can replay
- A gate that blocked the right moveGated-execute · Haiku 4.5
- Three proposals beat one agentPropose-select · Haiku 4.5
- A verifier that earned its keepPlan-execute-verify · Sonnet 5.5
- A revision that did not helpPlan-execute-verify · Haiku 4.5
- The control, missing a stepSingle · Haiku 4.5
Record this cell yourself: npm run bench -- run --strategies propose-select --tasks research-script-loading --models claude-sonnet-5-5 --repeats 1
Score breakdown
- passStates that both async and defer let a classic script be fetched in parallel with HTML parsing, so the fetch does not block the parser, unlike a classic script with src and neither attribute.1.00weight 0.13
met, backed by verified evidence E1, E2, E3. “`async` and `defer` both make an external script download in parallel with HTML parsing.”
- passStates that a defer script executes only after the document has finished parsing, and before DOMContentLoaded fires.1.00weight 0.19
met, backed by verified evidence E3, E9. “`defer` runs it after parsing finishes, in document order, before `DOMContentLoaded`.”
- passStates that an async script executes as soon as it is available, interrupting parsing if parsing is still in progress, so it may run before or after DOMContentLoaded.1.00weight 0.19
met, backed by verified evidence E2, E10. “An async script that arrives early can run before parsing ends, and so before `DOMContentLoaded`. A slow one runs after it.”
- passStates that multiple defer scripts execute in document order, while async scripts execute in the order their downloads complete, which is not predictable.1.00weight 0.19
met, backed by verified evidence E12, E13. “`defer`: document order.** MDN says "Scripts with the defer attribute will execute in the order in which they appear in the document."”
- partialStates that async and defer have no effect on a classic inline script (a script element without a src attribute).0.50weight 0.13
partial, backed by verified evidence E14. “MDN says that for inline classic scripts the attribute "would have no effect."”
- passStates that module scripts are deferred by default, so the defer attribute has no effect on them, while the async attribute does apply to module scripts.1.00weight 0.19
met, backed by verified evidence E4, E16. “The module and all its dependencies are fetched in parallel and run as soon as they are available, possibly before parsing completes.”
Penalties
No penalties were applied.
Scorer metadata
- corpus
- {"version":2,"snapshot":"2026-10-01","hash":"a5d703ba820ee9c957565ac15b9f98daeac8212372d73d7658a7a6bb2bbc1978","documents":6}
- opened documents
- ["mdn-script-element","whatwg-html-scripting","mdn-domcontentloaded"]
- opened by role
- {"secondary":["mdn-script-element","mdn-domcontentloaded"],"primary":["whatwg-html-scripting"]}
- displayed share
- {"mdn-script-element":0.8439,"whatwg-html-scripting":0.4616,"mdn-domcontentloaded":1,"webdev-third-party-javascript":0.0309,"webdev-optimize-resource-loading":0.0337,"mdn-js-modules":0.0055}
- cited count
- 3
- verified count
- 3
- hallucinated count
- 0
- citation integrity
- 1
- hallucinated urls
- []
- evidence block
- true
- evidence block repaired
- false
- evidence count
- 18
- verified quote count
- 18
- verified quote rate
- 1
- evidence
- [{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"A plain external classic script blocks parsing while it is fetched and evaluated","quote":"If neither attribute is present, then the script is fetched and evaluated immediately, blocking parsing until these are both complete.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"async classic scripts are fetched in parallel and run as soon as available, possibly before parsing completes","quote":"then the classic script will be fetched in parallel to parsing and evaluated as soon as it is available (potentially before parsing completes).","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"defer classic scripts are fetched in parallel and run once parsing has finished","quote":"If the async attribute is not present but the defer attribute is present, then the classic script will be fetched in parallel and evaluated when the page has finished parsing.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Module scripts are deferred by default and defer has no effect on them","quote":"(The defer attribute has no effect on module scripts.)","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Module scripts wait for parsing to finish regardless of where they appear in the document","quote":"no matter where this script element appears in the document, it will not be evaluated until both document parsing has complete","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Parser-inserted deferred and module scripts share the same post-parsing ordered list","quote":"Otherwise, if el has a defer attribute or el's type is \"module\":","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Setting both async and defer is allowed so legacy defer-only browsers fall back to deferred behaviour","quote":"The defer attribute may be specified even if the async attribute is specified, to cause legacy web browsers that only support defer (and not async) to fall back to the defer behavior instead of the blocking behavior that is the default.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"With both attributes present, async takes precedence","quote":"If the attribute is specified with the async attribute, the element will act as if only the async attribute is specified.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"defer scripts run before DOMContentLoaded and hold it up until they finish","quote":"Scripts with the defer attribute will prevent the DOMContentLoaded event from firing until the script has loaded and finished evaluating.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event","claim":"DOMContentLoaded waits for deferred and module scripts but not async scripts","quote":"It doesn't wait for other things like images, subframes, and async scripts to finish loading.","verified":true,"document":"mdn-domcontentloaded"},{"url":"https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event","claim":"MDN's event page says DOMContentLoaded fires before async scripts execute (the spec instead says async may run before parsing completes)","quote":"DOMContentLoaded is fired after the execution of these scripts, but before the execution of scripts with async.","verified":true,"document":"mdn-domcontentloaded"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"Deferred scripts execute in document order","quote":"Scripts with the defer attribute will execute in the order in which they appear in the document.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"async scripts have no ordering guarantee","quote":"You get no guarantee that scripts will run in any specific order.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"async has no effect on inline classic scripts","quote":"This attribute must not be used if the src attribute is absent (i.e., for inline scripts) for classic scripts, in this case it would have no effect.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"Inline non-module scripts run immediately during parsing","quote":"Scripts without async, defer or type=\"module\" attributes, as well as inline scripts without the type=\"module\" attribute, are fetched and executed immediately before the browser continues to parse the page.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"async on module scripts fetches the module and its dependencies in parallel and runs it as soon as available","quote":"For module scripts, if the async attribute is present then the scripts and all their dependencies will be fetched in parallel to parsing and evaluated as soon as they are available.","verified":true,"document":"mdn-script-element"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Parser-inserted scripts have force async false, while dynamically created scripts default to force async true","quote":"A script element has a force async boolean, initially true. It is set to false by the HTML parser and the XML parser on script elements they insert, and when the element gets an async content attribute added.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Inline module scripts are not run synchronously; they are queued as a task even with no dependencies","quote":"Queueing a task here means that, even if the inline module script has no dependencies or synchronously results in a parse error, we won't proceed to execute the script element synchronously.","verified":true,"document":"whatwg-html-scripting"}]
- judge unverified quotes
- 0
- judge verdicts
- [{"parallel-download":{"status":"met","support":["E1","E2","E3"]},"defer-timing":{"status":"met","support":["E3","E9"]},"async-timing":{"status":"met","support":["E2","E10"]},"ordering":{"status":"met","support":["E12","E13"]},"inline-ignored":{"status":"partial","support":["E14"]},"modules":{"status":"met","support":["E4","E16"]}},{"parallel-download":{"status":"met","support":["E1","E2","E3"]},"defer-timing":{"status":"met","support":["E3","E9"]},"async-timing":{"status":"met","support":["E2","E10"]},"ordering":{"status":"met","support":["E12","E13"]},"inline-ignored":{"status":"partial","support":["E14"]},"modules":{"status":"met","support":["E4","E16"]}},{"parallel-download":{"status":"met","support":["E1","E2","E3"]},"defer-timing":{"status":"met","support":["E3","E9"]},"async-timing":{"status":"met","support":["E2","E10"]},"ordering":{"status":"met","support":["E12","E13"]},"inline-ignored":{"status":"partial","support":["E14"]},"modules":{"status":"met","support":["E4","E16"]}}]
- judge failures
- 0
- grounded weight share
- 1
Final answer
async and defer both make an external script download in parallel with HTML parsing. They differ in when the script runs and whether order is guaranteed. async runs the script as soon as it has arrived, in no guaranteed order. defer runs it after parsing finishes, in document order, before DOMContentLoaded.
What each does
No attribute. A plain external classic script is fetched and run immediately, and the parser stops until both are done. The spec says: "If neither attribute is present, then the script is fetched and evaluated immediately, blocking parsing until these are both complete." (https://html.spec.whatwg.org/multipage/scripting.html)
async. For an external classic script, "the classic script will be fetched in parallel to parsing and evaluated as soon as it is available (potentially before parsing completes)." (https://html.spec.whatwg.org/multipage/scripting.html)
- Downloading doesn't block the parser, but execution does interrupt it. MDN says the executing script "blocks the page from rendering" (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script).
- In the spec's processing model, an async script goes into a "set of scripts that will execute as soon as possible". It is a set, so nothing orders its entries.
defer. The classic script "will be fetched in parallel and evaluated when the page has finished parsing." (https://html.spec.whatwg.org/multipage/scripting.html)
- The spec appends such scripts to a "list of scripts that will execute when the document has finished parsing", and the parser runs them from that list.
- MDN describes the timing as after the document has been parsed but before
DOMContentLoadedfires (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script).
Both together. async wins. MDN says "the element will act as if only the async attribute is specified." The spec says authors may still write both so that legacy browsers that only know defer fall back to deferred behaviour instead of blocking. (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script and https://html.spec.whatwg.org/multipage/scripting.html)
Timing relative to DOMContentLoaded
deferscripts hold upDOMContentLoaded. MDN's event page says it fires once the document is parsed "and all deferred scripts (<script defer src="…"> and <script type="module">) have downloaded and executed." It also says the event "doesn't wait for other things like images, subframes, and async scripts to finish loading." (https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event)- The sequence for deferred scripts.
document.readyStatebecomes "interactive" when parsing completes, before the deferred and module scripts run.DOMContentLoadedthen fires after they finish. (same MDN event page) asyncscripts have no fixed position.DOMContentLoadeddoesn't wait for them. An async script that arrives early can run before parsing ends, and so beforeDOMContentLoaded. A slow one runs after it. Code in an async script should checkdocument.readyStateinstead of assuming either case.- A difference in wording between the sources. The spec says an async script may run "potentially before parsing completes". A note on MDN's event page says
DOMContentLoadedfires "before the execution of scripts with async." The spec's wording is the broader one. Treat the MDN note as describing the common case, not a guarantee.
Execution order with several scripts
async: no ordering guarantee. MDN says "You get no guarantee that scripts will run in any specific order." Its example of three async scripts warns that a script depending on jQuery may run before jQuery has loaded. Useasynconly for independent scripts. (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script)defer: document order. MDN says "Scripts with the defer attribute will execute in the order in which they appear in the document." This holds even if a later script finishes downloading first, because the spec runs them from an ordered list. (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script)- Dynamically inserted scripts. Scripts created with
document.createElementhave "force async" set to true, so they behave as async unless you setscript.async = false. In that case they go into a separate in-order list and run in insertion order, without waiting for parsing to finish. (https://html.spec.whatwg.org/multipage/scripting.html)
Inline scripts
async and defer apply only to external classic scripts.
- MDN says that for inline classic scripts the attribute "would have no effect." (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script)
- A plain inline classic script runs immediately when the parser reaches it, and it blocks parsing. MDN says such scripts "are fetched and executed immediately before the browser continues to parse the page."
type="module" scripts
- Deferred by default. The spec says "Otherwise, the module script and its dependencies will be fetched in parallel to parsing and evaluated when the page has finished parsing. (The defer attribute has no effect on module scripts.)" (https://html.spec.whatwg.org/multipage/scripting.html)
- Same ordered list as
defer. In the spec's algorithm, a parser-inserted script goes into the post-parsing ordered list if it "has a defer attribute or el's type is "module"". Module scripts therefore run in document order, beforeDOMContentLoaded, alongside deferred classic scripts. Placement in the document doesn't change this. asyncon modules. The module and all its dependencies are fetched in parallel and run as soon as they are available, possibly before parsing completes. There is no ordering guarantee, as with async classic scripts.- Inline modules are the exception to the inline rule. The spec's attribute table allows
asyncon inline module scripts. Withoutasync, an inline module is deferred. Withasync, it goes into the as-soon-as-possible set. Unlike an inline classic script, it never runs synchronously at parse time. The spec says it always waits for a queued task, even when it has no dependencies.
{"evidence": [
{"claim": "A plain external classic script blocks parsing while it is fetched and evaluated", "quote": "If neither attribute is present, then the script is fetched and evaluated immediately, blocking parsing until these are both complete.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "async classic scripts are fetched in parallel and run as soon as available, possibly before parsing completes", "quote": "then the classic script will be fetched in parallel to parsing and evaluated as soon as it is available (potentially before parsing completes).", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "defer classic scripts are fetched in parallel and run once parsing has finished", "quote": "If the async attribute is not present but the defer attribute is present, then the classic script will be fetched in parallel and evaluated when the page has finished parsing.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Module scripts are deferred by default and defer has no effect on them", "quote": "(The defer attribute has no effect on module scripts.)", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Module scripts wait for parsing to finish regardless of where they appear in the document", "quote": "no matter where this script element appears in the document, it will not be evaluated until both document parsing has complete", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Parser-inserted deferred and module scripts share the same post-parsing ordered list", "quote": "Otherwise, if el has a defer attribute or el's type is \"module\":", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Setting both async and defer is allowed so legacy defer-only browsers fall back to deferred behaviour", "quote": "The defer attribute may be specified even if the async attribute is specified, to cause legacy web browsers that only support defer (and not async) to fall back to the defer behavior instead of the blocking behavior that is the default.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "With both attributes present, async takes precedence", "quote": "If the attribute is specified with the async attribute, the element will act as if only the async attribute is specified.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "defer scripts run before DOMContentLoaded and hold it up until they finish", "quote": "Scripts with the defer attribute will prevent the DOMContentLoaded event from firing until the script has loaded and finished evaluating.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "DOMContentLoaded waits for deferred and module scripts but not async scripts", "quote": "It doesn't wait for other things like images, subframes, and async scripts to finish loading.", "url": "https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event"},
{"claim": "MDN's event page says DOMContentLoaded fires before async scripts execute (the spec instead says async may run before parsing completes)", "quote": "DOMContentLoaded is fired after the execution of these scripts, but before the execution of scripts with async.", "url": "https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event"},
{"claim": "Deferred scripts execute in document order", "quote": "Scripts with the defer attribute will execute in the order in which they appear in the document.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "async scripts have no ordering guarantee", "quote": "You get no guarantee that scripts will run in any specific order.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "async has no effect on inline classic scripts", "quote": "This attribute must not be used if the src attribute is absent (i.e., for inline scripts) for classic scripts, in this case it would have no effect.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "Inline non-module scripts run immediately during parsing", "quote": "Scripts without async, defer or type=\"module\" attributes, as well as inline scripts without the type=\"module\" attribute, are fetched and executed immediately before the browser continues to parse the page.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "async on module scripts fetches the module and its dependencies in parallel and runs it as soon as available", "quote": "For module scripts, if the async attribute is present then the scripts and all their dependencies will be fetched in parallel to parsing and evaluated as soon as they are available.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "Parser-inserted scripts have force async false, while dynamically created scripts default to force async true", "quote": "A script element has a force async boolean, initially true. It is set to false by the HTML parser and the XML parser on script elements they insert, and when the element gets an async content attribute added.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Inline module scripts are not run synchronously; they are queued as a task even with no dependencies", "quote": "Queueing a task here means that, even if the inline module script has no dependencies or synchronously results in a parse error, we won't proceed to execute the script element synchronously.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"}
]}Provenance of the numbers on this page
- run set
- scaffold-or-scale-v1
- models
- Haiku 4.5, Sonnet 5.5, Opus 5.5
- effort
- high
- repeats
- 2
- fixtures
- replay
- runs
- 900 / 900
- strategy cost
- $354.44
- judge cost
- $9.19
- wall clock
- 9h 48m
- git
- fce5807b2d
Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.