ScaffoldBench

RunHaiku 4.5repeat 2

Orchestrator-workers on async vs defer on <script>

claude-haiku-4-5-20251001__orchestrator-workers__research-script-loading__r2

Score

0.906

wrong answer
design ran as designed
Cost
$0.113
+ $0.073 judging
Latency
1m 36s
setup 0.0s, grading 4.4s
Turns
15
Tool calls
11
Subagents
1
Tokens in / out
144k / 9.1k
107,675 read from cache

Did the design run?

every check held
  • Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
  • No API failures: heldharness.apiNo model call failed at the API.
  • Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
  • No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
  • Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
  • No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
  • Builtin tools: heldharness.builtin-toolsA check recorded with the run.
  • Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
  • No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
  • Judge answered: heldharness.judgeAt least one grading call returned a usable verdict.
  • The task was decomposed: heldorchestrator-workers.decompositionThe orchestrator’s reply was read through its schema and held at least one subtask.
  • The fan-out happened: heldorchestrator-workers.fan-outEvery subtask got a worker, every worker ran, and the recorded subagent count matches.
  • The synthesis wrote the answer: heldorchestrator-workers.synthesisThe final answer came from the synthesis call, not from a worker’s report standing in for it.
  • Each call was shown the right tools: heldorchestrator-workers.toolsThe orchestrator and the synthesis were shown no tools and every worker the task’s full toolset.

Phases

Decompose: 0.0s to 24.9s, finished, 0 tool calls, $0.016
Worker 1: all: 24.9s to 1m 19s, finished, 11 tool calls, $0.080
Synthesize: 1m 19s to 1m 36s, finished, 0 tool calls, $0.016

Drawn from the phase records: timing and cost per call, no tool calls or messages.

PhaseDurationCostTurnsToolsModel
Decompose24.9s$0.01620claude-haiku-4-5-20251001
Worker 1: all54.4s$0.0801211claude-haiku-4-5-20251001
Synthesize16.9s$0.01610claude-haiku-4-5-20251001

The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.

Runs you can replay

Record this cell yourself: npm run bench -- run --strategies orchestrator-workers --tasks research-script-loading --models claude-haiku-4-5-20251001 --repeats 1

Score breakdown

  • passStates that both async and defer let a classic script be fetched in parallel with HTML parsing, so the fetch does not block the parser, unlike a classic script with src and neither attribute.
    1.00weight 0.13

    met, backed by verified evidence E1, E10. “Both `async` and `defer` allow the HTML parser to continue while scripts download.”

  • passStates that a defer script executes only after the document has finished parsing, and before DOMContentLoaded fires.
    1.00weight 0.19

    met, backed by verified evidence E2, E4. “postpone execution until after the HTML document has been completely parsed”

  • partialStates that an async script executes as soon as it is available, interrupting parsing if parsing is still in progress, so it may run before or after DOMContentLoaded.
    0.50weight 0.19

    partial, backed by verified evidence E1, E5. “once the script finishes downloading, it executes immediately, regardless of whether the HTML parser has finished parsing the document. This execution can interrupt HTML parsing.”

  • passStates that multiple defer scripts execute in document order, while async scripts execute in the order their downloads complete, which is not predictable.
    1.00weight 0.19

    met, backed by verified evidence E3, E5. “if you have multiple deferred scripts, they execute in the order they appear in the markup”

  • passStates that async and defer have no effect on a classic inline script (a script element without a src attribute).
    1.00weight 0.13

    met, backed by verified evidence E6. “The `async` attribute has no effect on inline scripts (scripts without the `src` attribute)”

  • passStates that module scripts are deferred by default, so the defer attribute has no effect on them, while the async attribute does apply to module scripts.
    1.00weight 0.19

    met, backed by verified evidence E7, E8, E9. “defer has no effect on module scripts since modules are deferred by default.”

Penalties

No penalties were applied.

Scorer metadata

corpus
{"version":2,"snapshot":"2026-10-01","hash":"a5d703ba820ee9c957565ac15b9f98daeac8212372d73d7658a7a6bb2bbc1978","documents":6}
opened documents
["mdn-script-element","mdn-domcontentloaded","webdev-optimize-resource-loading","mdn-js-modules","webdev-third-party-javascript"]
opened by role
{"secondary":["mdn-script-element","mdn-domcontentloaded","webdev-optimize-resource-loading","mdn-js-modules","webdev-third-party-javascript"]}
displayed share
{"mdn-script-element":0.8439,"webdev-third-party-javascript":0.9618,"whatwg-html-scripting":0.0116,"webdev-optimize-resource-loading":0.9624,"mdn-domcontentloaded":1,"mdn-js-modules":0.1744}
cited count
4
verified count
4
hallucinated count
0
citation integrity
1
hallucinated urls
[]
evidence block
true
evidence block repaired
false
evidence count
10
verified quote count
10
verified quote rate
1
evidence
[{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"For classic scripts, async allows the script to be fetched in parallel to parsing and evaluated as soon as it is available.","quote":"For classic scripts, if the async attribute is present, then the classic script will be fetched in parallel to parsing and evaluated as soon as it is available.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"The defer attribute indicates the script should execute after the document has been parsed but before the DOMContentLoaded event.","quote":"This Boolean attribute is set to indicate to a browser that the script is meant to be executed after the document has been parsed, but before firing DOMContentLoaded event.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"Scripts with defer execute in the order they appear in the document.","quote":"Scripts with the defer attribute will execute in the order in which they appear in the document.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event","claim":"The DOMContentLoaded event fires after deferred scripts execute but before async scripts.","quote":"The DOMContentLoaded event fires when the HTML document has been completely parsed, and all deferred scripts (<script defer src=\"…\"> and <script type=\"module\">) have downloaded and executed. It doesn't wait for other things like images, subframes, and async scripts to finish loading.","verified":true,"document":"mdn-domcontentloaded"},{"url":"https://web.dev/articles/efficiently-load-third-party-javascript","claim":"Async scripts may execute out of order and can interrupt DOM building.","quote":"This means it's possible (and likely) that async scripts won't run in the order in which they appear in the HTML. It also means they can interrupt DOM building if they finish downloading while the parser is still at work.","verified":true,"document":"webdev-third-party-javascript"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"Async and defer attributes have no effect on inline scripts.","quote":"This attribute must not be used if the src attribute is absent (i.e., for inline scripts) for classic scripts, in this case it would have no effect.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules","claim":"Module scripts are deferred by default and do not require the defer attribute.","quote":"There is no need to use the defer attribute (see <script> attributes) when loading a module script; modules are deferred automatically.","verified":true,"document":"mdn-js-modules"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"Module scripts have no effect from defer and charset attributes.","quote":"The processing of the script contents is deferred. The charset and defer attributes have no effect.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"For module scripts, async causes them to execute as soon as available rather than being deferred.","quote":"For module scripts, if the async attribute is present then the scripts and all their dependencies will be fetched in parallel to parsing and evaluated as soon as they are available.","verified":true,"document":"mdn-script-element"},{"url":"https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script","claim":"Scripts without async, defer, or type=module execute immediately and block parsing.","quote":"Scripts without async, defer or type=\"module\" attributes, as well as inline scripts without the type=\"module\" attribute, are fetched and executed immediately before the browser continues to parse the page.","verified":true,"document":"mdn-script-element"}]
judge unverified quotes
0
judge verdicts
[{"parallel-download":{"status":"met","support":["E1","E10"]},"defer-timing":{"status":"met","support":["E2","E4"]},"async-timing":{"status":"partial","support":["E1","E5"]},"ordering":{"status":"met","support":["E3","E5"]},"inline-ignored":{"status":"met","support":["E6"]},"modules":{"status":"met","support":["E7","E8","E9"]}},{"parallel-download":{"status":"met","support":["E1","E10"]},"defer-timing":{"status":"met","support":["E2","E4"]},"async-timing":{"status":"partial","support":["E1","E5"]},"ordering":{"status":"met","support":["E3","E5"]},"inline-ignored":{"status":"met","support":["E6"]},"modules":{"status":"met","support":["E7","E8","E9"]}},{"parallel-download":{"status":"met","support":["E1","E10"]},"defer-timing":{"status":"met","support":["E2","E4"]},"async-timing":{"status":"partial","support":["E1","E5"]},"ordering":{"status":"met","support":["E3","E5"]},"inline-ignored":{"status":"met","support":["E6"]},"modules":{"status":"met","support":["E7","E8","E9"]}}]
judge failures
0
grounded weight share
1

Final answer

The async and defer Attributes on HTML <script> Elements

The async and defer attributes control how external scripts are fetched and executed relative to HTML parsing, but they have fundamentally different behaviors.

The async Attribute

When the async attribute is present on a classic script element with a src attribute, the browser downloads the script in parallel with HTML parsing. However, once the script finishes downloading, it executes immediately, regardless of whether the HTML parser has finished parsing the document. This execution can interrupt HTML parsing. Because scripts download independently and can finish at unpredictable times, async scripts are not guaranteed to execute in any particular order—they may run out of order relative to how they appear in the markup. The DOMContentLoaded event does not wait for async scripts to complete.

The async attribute has no effect on inline scripts (scripts without the src attribute), as inline scripts are always executed immediately. For type="module" scripts, if the async attribute is present, they execute as soon as available rather than being deferred by default.

The defer Attribute

The defer attribute instructs the browser to download the script in parallel with HTML parsing but postpone execution until after the HTML document has been completely parsed. Critically, defer scripts maintain their execution order relative to the document—if you have multiple deferred scripts, they execute in the order they appear in the markup. The DOMContentLoaded event fires after all deferred scripts have downloaded and executed.

Like async, the defer attribute has no effect on inline scripts and cannot be used on them. Additionally, defer has no effect on module scripts since modules are deferred by default.

Module Scripts and Inline Module Scripts

Classic <script type="module"> elements are deferred automatically by default—there is no need to add the defer attribute. They behave like scripts with defer: they load in parallel but execute after HTML parsing completes, in document order, and before DOMContentLoaded fires. However, if the async attribute is present on a module script, it behaves like an async script instead, executing as soon as it's available.

Inline scripts with type="module" are also deferred automatically and execute in-place in the document order. Standard inline scripts (without type="module") cannot use async or defer; they execute synchronously and block HTML parsing.

Interaction with HTML Parsing

Classic scripts without async or defer block the HTML parser entirely until they download and execute. Both async and defer allow the HTML parser to continue while scripts download. However, async scripts can interrupt the parser when they finish downloading and begin execution. Defer scripts never interrupt parsing—they wait until parsing is completely finished.

Execution Order Guarantees

  • No attributes (classic scripts): Execute immediately in document order, blocking the parser
  • async: Execute as soon as downloaded; order is unpredictable
  • defer: Execute in document order, after parsing completes
  • type="module": Execute in document order after parsing (as if defer), unless async is also present

{"evidence": [{"claim": "For classic scripts, async allows the script to be fetched in parallel to parsing and evaluated as soon as it is available.", "quote": "For classic scripts, if the async attribute is present, then the classic script will be fetched in parallel to parsing and evaluated as soon as it is available.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"}, {"claim": "The defer attribute indicates the script should execute after the document has been parsed but before the DOMContentLoaded event.", "quote": "This Boolean attribute is set to indicate to a browser that the script is meant to be executed after the document has been parsed, but before firing DOMContentLoaded event.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"}, {"claim": "Scripts with defer execute in the order they appear in the document.", "quote": "Scripts with the defer attribute will execute in the order in which they appear in the document.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"}, {"claim": "The DOMContentLoaded event fires after deferred scripts execute but before async scripts.", "quote": "The DOMContentLoaded event fires when the HTML document has been completely parsed, and all deferred scripts (<script defer src=\"…\"> and <script type=\"module\">) have downloaded and executed. It doesn't wait for other things like images, subframes, and async scripts to finish loading.", "url": "https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event"}, {"claim": "Async scripts may execute out of order and can interrupt DOM building.", "quote": "This means it's possible (and likely) that async scripts won't run in the order in which they appear in the HTML. It also means they can interrupt DOM building if they finish downloading while the parser is still at work.", "url": "https://web.dev/articles/efficiently-load-third-party-javascript"}, {"claim": "Async and defer attributes have no effect on inline scripts.", "quote": "This attribute must not be used if the src attribute is absent (i.e., for inline scripts) for classic scripts, in this case it would have no effect.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"}, {"claim": "Module scripts are deferred by default and do not require the defer attribute.", "quote": "There is no need to use the defer attribute (see <script> attributes) when loading a module script; modules are deferred automatically.", "url": "https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules"}, {"claim": "Module scripts have no effect from defer and charset attributes.", "quote": "The processing of the script contents is deferred. The charset and defer attributes have no effect.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"}, {"claim": "For module scripts, async causes them to execute as soon as available rather than being deferred.", "quote": "For module scripts, if the async attribute is present then the scripts and all their dependencies will be fetched in parallel to parsing and evaluated as soon as they are available.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"}, {"claim": "Scripts without async, defer, or type=module execute immediately and block parsing.", "quote": "Scripts without async, defer or type=\"module\" attributes, as well as inline scripts without the type=\"module\" attribute, are fetched and executed immediately before the browser continues to parse the page.", "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script"}]}

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.