ScaffoldBench

RunOpus 5.5repeat 1

Propose-select on async vs defer on <script>

claude-opus-5-5__propose-select__research-script-loading__r1

Score

1.000

passed
design ran as designed
Cost
$1.30
+ $0.082 judging
Latency
2m 18s
setup 0.0s, grading 6.0s
Turns
52
Tool calls
47
Subagents
3
Tokens in / out
587k / 25k
487,015 read from cache

Did the design run?

every check held
  • Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
  • No API failures: heldharness.apiNo model call failed at the API.
  • Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
  • No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
  • Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
  • No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
  • Builtin tools: heldharness.builtin-toolsA check recorded with the run.
  • Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
  • No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
  • Judge answered: heldharness.judgeAt least one grading call returned a usable verdict.
  • All three proposers ran: heldpropose-select.proposalsThree proposers ran in parallel, each from its own angle, and none was starved while another ran.
  • The selector chose: heldpropose-select.selectionThe selector picked a proposal by a valid number, read from its schema reply and not a default.
  • Each call was shown the right tools: heldpropose-select.toolsEach proposer was shown the read-only tools, the selector none, and the executor the full toolset.

Phases

Proposal 1 (policy-first): 0.0s to 58.2s, finished, 14 tool calls, $0.329
Proposal 2 (state-first): 0.0s to 1m 36s, finished, 18 tool calls, $0.397
Proposal 3 (outcome-first): 0.0s to 1m 5s, finished, 15 tool calls, $0.357
Select: 1m 36s to 2m 18s, finished, 0 tool calls, $0.218

Drawn from the phase records: timing and cost per call, no tool calls or messages.

PhaseDurationCostTurnsToolsModel
Proposal 1 (policy-first)58.2s$0.3291514claude-opus-5-5
Proposal 2 (state-first)1m 36s$0.3971918claude-opus-5-5
Proposal 3 (outcome-first)1m 5s$0.3571615claude-opus-5-5
Select41.2s$0.21820claude-opus-5-5

The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.

Runs you can replay

Record this cell yourself: npm run bench -- run --strategies propose-select --tasks research-script-loading --models claude-opus-5-5 --repeats 1

Score breakdown

  • passStates that both async and defer let a classic script be fetched in parallel with HTML parsing, so the fetch does not block the parser, unlike a classic script with src and neither attribute.
    1.00weight 0.13

    met, backed by verified evidence E1, E2, E3. “The browser downloads the script while it keeps parsing, then runs it as soon as the download finishes.”

  • passStates that a defer script executes only after the document has finished parsing, and before DOMContentLoaded fires.
    1.00weight 0.19

    met, backed by verified evidence E6, E9, E3. “it runs only after parsing has finished and before DOMContentLoaded fires”

  • passStates that an async script executes as soon as it is available, interrupting parsing if parsing is still in progress, so it may run before or after DOMContentLoaded.
    1.00weight 0.19

    met, backed by verified evidence E2, E4. “runs it as soon as the download finishes. That can happen before parsing is done, so running it can briefly pause the parser.”

  • passStates that multiple defer scripts execute in document order, while async scripts execute in the order their downloads complete, which is not predictable.
    1.00weight 0.19

    met, backed by verified evidence E8, E5, E19. “deferred scripts run in the order they appear in the document.”

  • passStates that async and defer have no effect on a classic inline script (a script element without a src attribute).
    1.00weight 0.13

    met, backed by verified evidence E14. “`async` and `defer` do nothing on them. MDN warns they must not be used when `src` is absent”

  • passStates that module scripts are deferred by default, so the defer attribute has no effect on them, while the async attribute does apply to module scripts.
    1.00weight 0.19

    met, backed by verified evidence E16, E17. “The `defer` attribute has no effect on them.”

Penalties

No penalties were applied.

Scorer metadata

corpus
{"version":2,"snapshot":"2026-10-01","hash":"a5d703ba820ee9c957565ac15b9f98daeac8212372d73d7658a7a6bb2bbc1978","documents":6}
opened documents
["mdn-script-element","whatwg-html-scripting","mdn-domcontentloaded","webdev-third-party-javascript","mdn-js-modules","webdev-optimize-resource-loading"]
opened by role
{"secondary":["mdn-script-element","mdn-domcontentloaded","webdev-third-party-javascript","mdn-js-modules","webdev-optimize-resource-loading"],"primary":["whatwg-html-scripting"]}
displayed share
{"mdn-script-element":0.8544,"whatwg-html-scripting":0.2642,"mdn-domcontentloaded":1,"webdev-optimize-resource-loading":0.3583,"webdev-third-party-javascript":1,"mdn-js-modules":0.1742}
cited count
4
verified count
4
hallucinated count
0
citation integrity
1
hallucinated urls
[]
evidence block
true
evidence block repaired
false
evidence count
20
verified quote count
20
verified quote rate
1
evidence
[{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"A classic external script with neither attribute is fetched and run immediately and blocks parsing.","quote":"If neither attribute is present, then the script is fetched and evaluated immediately, blocking parsing until these are both complete.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"An async classic script downloads while parsing continues and runs as soon as it is available, possibly before parsing finishes.","quote":"For external classic scripts, if the async attribute is present, then the classic script will be fetched in parallel to parsing and evaluated as soon as it is available (potentially before parsing completes).","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"A deferred classic script downloads in parallel and runs when parsing has finished.","quote":"If the async attribute is not present but the defer attribute is present, then the classic script will be fetched in parallel and evaluated when the page has finished parsing.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://web.dev/articles/efficiently-load-third-party-javascript","claim":"Async scripts can interrupt DOM building.","quote":"It also means they can interrupt DOM building if they finish downloading while the parser is still at work.","verified":true,"document":"webdev-third-party-javascript"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"Async scripts have no guaranteed execution order.","quote":"You get no guarantee that scripts will run in any specific order.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"Deferred scripts run after parsing and before DOMContentLoaded.","quote":"This Boolean attribute is set to indicate to a browser that the script is meant to be executed after the document has been parsed, but before firing DOMContentLoaded event.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"Deferred scripts hold DOMContentLoaded back until they finish.","quote":"Scripts with the defer attribute will prevent the DOMContentLoaded event from firing until the script has loaded and finished evaluating.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"Deferred scripts run in document order.","quote":"Scripts with the defer attribute will execute in the order in which they appear in the document.","verified":true,"document":"mdn-script-element"},{"url":"https://web.dev/articles/efficiently-load-third-party-javascript","claim":"web.dev agrees that defer runs after parsing, before DOMContentLoaded.","quote":"Scripts with the defer attribute execute after HTML parsing is completely finished, but before the DOMContentLoaded event.","verified":true,"document":"webdev-third-party-javascript"},{"url":"https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event","claim":"DOMContentLoaded waits for deferred and module scripts but not async ones.","quote":"The DOMContentLoaded event fires when the HTML document has been completely parsed, and all deferred scripts (<script defer src=\"…\"> and <script type=\"module\">) have downloaded and executed.","verified":true,"document":"mdn-domcontentloaded"},{"url":"https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event","claim":"MDN's DOMContentLoaded page says async scripts run after the event, which conflicts with the spec's 'potentially before parsing completes'.","quote":"DOMContentLoaded is fired after the execution of these scripts, but before the execution of scripts with async.","verified":true,"document":"mdn-domcontentloaded"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"If both attributes are present, async wins.","quote":"If the attribute is specified with the defer attribute, the element will act as if only the async attribute is specified.","verified":true,"document":"mdn-script-element"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Authors add defer next to async as a fallback for old browsers.","quote":"The defer attribute may be specified even if the async attribute is specified, to cause legacy web browsers that only support defer (and not async) to fall back to the defer behavior instead of the blocking behavior that is the default.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"defer has no effect on inline classic scripts.","quote":"Warning: This attribute must not be used if the src attribute is absent (i.e., for inline scripts), in this case it would have no effect.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"Inline classic scripts run immediately, before parsing continues.","quote":"Scripts without async, defer or type=\"module\" attributes, as well as inline scripts without the type=\"module\" attribute, are fetched and executed immediately before the browser continues to parse the page.","verified":true,"document":"mdn-script-element"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Module scripts without async are deferred, and the defer attribute has no effect on them.","quote":"Otherwise, the module script and its dependencies will be fetched in parallel to parsing and evaluated when the page has finished parsing. (The defer attribute has no effect on module scripts.)","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Async module scripts load with all their dependencies in parallel and may run before parsing finishes.","quote":"For module scripts, if the async attribute is present, then the module script and all its dependencies will be fetched in parallel to parsing, and the module script will be evaluated as soon as it is available (potentially before parsing completes).","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Inline module scripts never run synchronously during parsing.","quote":"Queueing a task here means that, even if the inline module script has no dependencies or synchronously results in a parse error, we won't proceed to execute the script element synchronously.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Deferred scripts and non-async modules go into an ordered list that runs when parsing finishes.","quote":"Each Document has a list of scripts that will execute when the document has finished parsing, which is a list of script elements, initially empty.","verified":true,"document":"whatwg-html-scripting"},{"url":"https://html.spec.whatwg.org/multipage/scripting.html","claim":"Scripts without src that aren't modules run immediately.","quote":"Otherwise, immediately execute the script element el, even if other scripts are already executing.","verified":true,"document":"whatwg-html-scripting"}]
judge unverified quotes
0
judge verdicts
[{"parallel-download":{"status":"met","support":["E1","E2","E3"]},"defer-timing":{"status":"met","support":["E6","E9","E3"]},"async-timing":{"status":"met","support":["E2","E4"]},"ordering":{"status":"met","support":["E8","E5","E19"]},"inline-ignored":{"status":"met","support":["E14"]},"modules":{"status":"met","support":["E16","E17"]}},{"parallel-download":{"status":"met","support":["E2","E3","E1"]},"defer-timing":{"status":"met","support":["E6","E9","E3"]},"async-timing":{"status":"met","support":["E2","E4"]},"ordering":{"status":"met","support":["E8","E5","E19"]},"inline-ignored":{"status":"met","support":["E14","E20"]},"modules":{"status":"met","support":["E16","E17"]}},{"parallel-download":{"status":"met","support":["E1","E2","E3"]},"defer-timing":{"status":"met","support":["E6","E9","E7"]},"async-timing":{"status":"met","support":["E2","E4"]},"ordering":{"status":"met","support":["E8","E5","E19"]},"inline-ignored":{"status":"met","support":["E14","E20"]},"modules":{"status":"met","support":["E16","E17"]}}]
judge failures
0
grounded weight share
1

Final answer

Here's how async and defer work. I relied mainly on the WHATWG HTML Standard's scripting section and checked it against MDN and web.dev.

What happens with no attribute. A plain external <script src> stops the parser. The spec says "If neither attribute is present, then the script is fetched and evaluated immediately, blocking parsing until these are both complete." (https://html.spec.whatwg.org/multipage/scripting.html). An inline classic <script>…</script> also runs right away during parsing. MDN says such scripts "are fetched and executed immediately before the browser continues to parse the page" (https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script).

async (classic script with src). The browser downloads the script while it keeps parsing, then runs it as soon as the download finishes. That can happen before parsing is done, so running it can briefly pause the parser. web.dev says async scripts "can interrupt DOM building if they finish downloading while the parser is still at work" (https://web.dev/articles/efficiently-load-third-party-javascript).

  • Order: none is guaranteed. MDN says "You get no guarantee that scripts will run in any specific order." The spec keeps these scripts in an unordered "set of scripts that will execute as soon as possible."
  • DOMContentLoaded: the event doesn't wait for async scripts (https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event). An async script can run before or after the event.
  • The sources disagree here. A note on MDN's DOMContentLoaded page says the event fires "before the execution of scripts with async." The spec says an async script is evaluated "as soon as it is available (potentially before parsing completes)", which can be before DOMContentLoaded. web.dev only promises async scripts run before the window's load event. I'd follow the spec: don't assume either order.

defer (classic script with src). The script also downloads while parsing continues, but it runs only after parsing has finished and before DOMContentLoaded fires (MDN's <script> page; web.dev).

  • Order: deferred scripts run in the order they appear in the document. In the spec they go into an ordered "list of scripts that will execute when the document has finished parsing."
  • DOMContentLoaded: MDN says deferred scripts "will prevent the DOMContentLoaded event from firing until the script has loaded and finished evaluating."
  • Wording conflict: a summary elsewhere on MDN's <script> page says deferred scripts run "once everything has finished loading," which sounds like the load event. The spec, MDN's attribute definition and web.dev all say "after parsing, before DOMContentLoaded," so that's the accurate version.

Both attributes together. async wins. MDN says the element "will act as if only the async attribute is specified". The spec explains that authors add both so that old browsers which only support defer fall back to deferring instead of blocking.

Inline classic scripts. async and defer do nothing on them. MDN warns they must not be used when src is absent, because "it would have no effect". In the spec's processing steps, the async/defer queues apply only to classic scripts that have a src, and to module scripts. Any other script falls through to "immediately execute the script element".

type="module" scripts.

  • Deferred by default. The defer attribute has no effect on them. Without async, a module and all its imports download while parsing continues, and the module runs after parsing ends. It joins the same ordered list as defer scripts, so modules keep document order and run before DOMContentLoaded. MDN's DOMContentLoaded page counts <script type="module"> among the deferred scripts the event waits for.
  • With async: the module runs as soon as it and all its dependencies have loaded. There's no ordering guarantee, and it may run before parsing finishes.
  • Inline modules: unlike classic scripts, async does work on inline modules (the spec's attribute table allows it). An inline module never runs synchronously during parsing, even if it has no imports.

Scripts added from JavaScript. Scripts inserted by other scripts behave as async by default. If you turn that flag off, they run in insertion order, using the spec's "list of scripts that will execute in order as soon as possible".

One gap: the spec's parsing chapter returned a 404, so the claim that deferred scripts run before DOMContentLoaded rests on MDN and web.dev, not the spec's own steps.

{"evidence": [
{"claim": "A classic external script with neither attribute is fetched and run immediately and blocks parsing.", "quote": "If neither attribute is present, then the script is fetched and evaluated immediately, blocking parsing until these are both complete.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "An async classic script downloads while parsing continues and runs as soon as it is available, possibly before parsing finishes.", "quote": "For external classic scripts, if the async attribute is present, then the classic script will be fetched in parallel to parsing and evaluated as soon as it is available (potentially before parsing completes).", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "A deferred classic script downloads in parallel and runs when parsing has finished.", "quote": "If the async attribute is not present but the defer attribute is present, then the classic script will be fetched in parallel and evaluated when the page has finished parsing.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Async scripts can interrupt DOM building.", "quote": "It also means they can interrupt DOM building if they finish downloading while the parser is still at work.", "url": "https://web.dev/articles/efficiently-load-third-party-javascript"},
{"claim": "Async scripts have no guaranteed execution order.", "quote": "You get no guarantee that scripts will run in any specific order.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "Deferred scripts run after parsing and before DOMContentLoaded.", "quote": "This Boolean attribute is set to indicate to a browser that the script is meant to be executed after the document has been parsed, but before firing DOMContentLoaded event.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "Deferred scripts hold DOMContentLoaded back until they finish.", "quote": "Scripts with the defer attribute will prevent the DOMContentLoaded event from firing until the script has loaded and finished evaluating.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "Deferred scripts run in document order.", "quote": "Scripts with the defer attribute will execute in the order in which they appear in the document.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "web.dev agrees that defer runs after parsing, before DOMContentLoaded.", "quote": "Scripts with the defer attribute execute after HTML parsing is completely finished, but before the DOMContentLoaded event.", "url": "https://web.dev/articles/efficiently-load-third-party-javascript"},
{"claim": "DOMContentLoaded waits for deferred and module scripts but not async ones.", "quote": "The DOMContentLoaded event fires when the HTML document has been completely parsed, and all deferred scripts (<script defer src=\"…\"> and <script type=\"module\">) have downloaded and executed.", "url": "https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event"},
{"claim": "MDN's DOMContentLoaded page says async scripts run after the event, which conflicts with the spec's 'potentially before parsing completes'.", "quote": "DOMContentLoaded is fired after the execution of these scripts, but before the execution of scripts with async.", "url": "https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event"},
{"claim": "If both attributes are present, async wins.", "quote": "If the attribute is specified with the defer attribute, the element will act as if only the async attribute is specified.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "Authors add defer next to async as a fallback for old browsers.", "quote": "The defer attribute may be specified even if the async attribute is specified, to cause legacy web browsers that only support defer (and not async) to fall back to the defer behavior instead of the blocking behavior that is the default.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "defer has no effect on inline classic scripts.", "quote": "Warning: This attribute must not be used if the src attribute is absent (i.e., for inline scripts), in this case it would have no effect.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "Inline classic scripts run immediately, before parsing continues.", "quote": "Scripts without async, defer or type=\"module\" attributes, as well as inline scripts without the type=\"module\" attribute, are fetched and executed immediately before the browser continues to parse the page.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"},
{"claim": "Module scripts without async are deferred, and the defer attribute has no effect on them.", "quote": "Otherwise, the module script and its dependencies will be fetched in parallel to parsing and evaluated when the page has finished parsing. (The defer attribute has no effect on module scripts.)", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Async module scripts load with all their dependencies in parallel and may run before parsing finishes.", "quote": "For module scripts, if the async attribute is present, then the module script and all its dependencies will be fetched in parallel to parsing, and the module script will be evaluated as soon as it is available (potentially before parsing completes).", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Inline module scripts never run synchronously during parsing.", "quote": "Queueing a task here means that, even if the inline module script has no dependencies or synchronously results in a parse error, we won't proceed to execute the script element synchronously.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Deferred scripts and non-async modules go into an ordered list that runs when parsing finishes.", "quote": "Each Document has a list of scripts that will execute when the document has finished parsing, which is a list of script elements, initially empty.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"},
{"claim": "Scripts without src that aren't modules run immediately.", "quote": "Otherwise, immediately execute the script element el, even if other scripts are already executing.", "url": "https://html.spec.whatwg.org/multipage/scripting.html"}
]}

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.