ScaffoldBench

Architecture

Orchestrator-workers

Does splitting a task across parallel contexts pay for the coordination?

An orchestrator splits the task, workers run the parts in parallel contexts, and a final call merges their reports.

How it runs

  1. 1The orchestrator

    no tools

    Splits the task into two to four independent subtasks through a structured reply, or one worker with the whole task when it judges the task cannot honestly be split.

  2. 2The workers

    full toolsetruns in parallel

    One worker per subtask, started together in separate contexts with the full toolset. They cannot see each other, so an overlap or a dependency the split missed shows up in the score.

  3. 3The synthesis

    no tools

    Writes the final answer from the workers’ reports. It holds no tools, so it can only merge what the workers found.

Where it can fail quietly

If the split never happens the design is one worker with an orchestrator around it, and the score looks like a single agent’s.

What the structure buys

Orchestrator-workers against a single agent on the same tasks and repeats, on each model. The interval is on the paired difference, so it is as wide as the evidence is thin.

  • Haiku 4.5=1 of 6
    Score
    0.701
    0.573–0.824
    Cost / run
    $0.171
    2.0× single
    Ranked
    50/50

    not separable from single−0.070 [−0.152, −0.001]too few tasks carry the spread to show equivalence

  • Sonnet 5.5=1 of 6
    Score
    0.924
    0.883–0.962
    Cost / run
    $0.216
    1.8× single
    Ranked
    50/50

    not separable from single+0.025 [−0.005, +0.078]too few tasks carry the spread to show equivalence

  • Opus 5.5=1 of 6
    Score
    0.921
    0.869–0.961
    Cost / run
    $0.459
    1.8× single
    Ranked
    50/50

    not separable from single−0.019 [−0.051, +0.006]too few tasks carry the spread to show equivalence

Task by task against a single agent

A mean can hide whether a design helps everywhere a little or on one task a lot. Each bar is the difference from a single agent on one task; the best-served tasks are at the top.

Haiku 4.5
Sonnet 5.5
Opus 5.5
Haiku 4.5: level with a single agent on Escalate a payroll outage the requester wants closed
Sonnet 5.5: +0.50 against a single agent on Escalate a payroll outage the requester wants closed
Opus 5.5: level with a single agent on Escalate a payroll outage the requester wants closed
Haiku 4.5: +0.29 against a single agent on Add checked bags and collect the right fee
Sonnet 5.5: level with a single agent on Add checked bags and collect the right fee
Opus 5.5: level with a single agent on Add checked bags and collect the right fee
Haiku 4.5: +0.07 against a single agent on Share two short deliveries out among backorders
Sonnet 5.5: +0.09 against a single agent on Share two short deliveries out among backorders
Opus 5.5: level with a single agent on Share two short deliveries out among backorders
Haiku 4.5: level with a single agent on Work a morning's customer inbox
Sonnet 5.5: level with a single agent on Work a morning's customer inbox
Opus 5.5: +0.12 against a single agent on Work a morning's customer inbox
Haiku 4.5: +0.10 against a single agent on OWASP vs NIST on password hashing
Sonnet 5.5: level with a single agent on OWASP vs NIST on password hashing
Opus 5.5: level with a single agent on OWASP vs NIST on password hashing
Haiku 4.5: level with a single agent on Consolidate duplicate CRM reports
Sonnet 5.5: +0.08 against a single agent on Consolidate duplicate CRM reports
Opus 5.5: level with a single agent on Consolidate duplicate CRM reports
Haiku 4.5: +0.01 against a single agent on Settle a desk of returns by what actually came back
Sonnet 5.5: level with a single agent on Settle a desk of returns by what actually came back
Opus 5.5: level with a single agent on Settle a desk of returns by what actually came back
Haiku 4.5: level with a single agent on Redirect the order that has not shipped
Sonnet 5.5: level with a single agent on Redirect the order that has not shipped
Opus 5.5: level with a single agent on Redirect the order that has not shipped
Haiku 4.5: level with a single agent on Cancel one line of a multi-line order
Sonnet 5.5: level with a single agent on Cancel one line of a multi-line order
Opus 5.5: level with a single agent on Cancel one line of a multi-line order
Haiku 4.5: level with a single agent on NIST SP 800-63B password rules, current revision
Sonnet 5.5: level with a single agent on NIST SP 800-63B password rules, current revision
Opus 5.5: level with a single agent on NIST SP 800-63B password rules, current revision
Haiku 4.5: level with a single agent on Triage a new service desk ticket
Sonnet 5.5: level with a single agent on Triage a new service desk ticket
Opus 5.5: level with a single agent on Triage a new service desk ticket
Haiku 4.5: +0.03 against a single agent on HTTP 301 vs 308
Sonnet 5.5: +0.06 against a single agent on HTTP 301 vs 308
Opus 5.5: −0.10 against a single agent on HTTP 301 vs 308
Haiku 4.5: −0.01 against a single agent on Work a cancellation queue where cash cannot be recalled
Sonnet 5.5: level with a single agent on Work a cancellation queue where cash cannot be recalled
Opus 5.5: level with a single agent on Work a cancellation queue where cash cannot be recalled
Haiku 4.5: −0.02 against a single agent on Triage the weekend's queue on Monday morning
Sonnet 5.5: level with a single agent on Triage the weekend's queue on Monday morning
Opus 5.5: level with a single agent on Triage the weekend's queue on Monday morning
Haiku 4.5: −0.05 against a single agent on Where ten HTTP header fields are defined
Sonnet 5.5: level with a single agent on Where ten HTTP header fields are defined
Opus 5.5: level with a single agent on Where ten HTTP header fields are defined
Haiku 4.5: −0.08 against a single agent on Re-protect a cancelled flight when seats are scarce
Sonnet 5.5: level with a single agent on Re-protect a cancelled flight when seats are scarce
Opus 5.5: level with a single agent on Re-protect a cancelled flight when seats are scarce
Haiku 4.5: level with a single agent on Twelve HTTP status codes, by the book
Sonnet 5.5: level with a single agent on Twelve HTTP status codes, by the book
Opus 5.5: −0.13 against a single agent on Twelve HTTP status codes, by the book
Haiku 4.5: −0.05 against a single agent on async vs defer on <script>
Sonnet 5.5: level with a single agent on async vs defer on <script>
Opus 5.5: −0.09 against a single agent on async vs defer on <script>
Haiku 4.5: −0.21 against a single agent on Run the major-incident runbook for a CI outage
Sonnet 5.5: level with a single agent on Run the major-incident runbook for a CI outage
Opus 5.5: level with a single agent on Run the major-incident runbook for a CI outage
Haiku 4.5: −0.21 against a single agent on Rebook onto the earliest available flight and settle the difference
Sonnet 5.5: level with a single agent on Rebook onto the earliest available flight and settle the difference
Opus 5.5: level with a single agent on Rebook onto the earliest available flight and settle the difference
Haiku 4.5: −0.29 against a single agent on Cancel a fare the customer wrongly believes is refundable to their card
Sonnet 5.5: level with a single agent on Cancel a fare the customer wrongly believes is refundable to their card
Opus 5.5: level with a single agent on Cancel a fare the customer wrongly believes is refundable to their card
Haiku 4.5: −0.01 against a single agent on Audit a week of baggage fees
Sonnet 5.5: −0.01 against a single agent on Audit a week of baggage fees
Opus 5.5: −0.27 against a single agent on Audit a week of baggage fees
Haiku 4.5: −0.37 against a single agent on Work the access request queue
Sonnet 5.5: level with a single agent on Work the access request queue
Opus 5.5: level with a single agent on Work the access request queue
Haiku 4.5: −0.30 against a single agent on Return an expired, partially shipped order
Sonnet 5.5: −0.09 against a single agent on Return an expired, partially shipped order
Opus 5.5: level with a single agent on Return an expired, partially shipped order
Haiku 4.5: −0.66 against a single agent on Settle change requests under dated bulletins
Sonnet 5.5: level with a single agent on Settle change requests under dated bulletins
Opus 5.5: level with a single agent on Settle change requests under dated bulletins
Mean score of the architecture minus a single agent’s, over the repeats both completed, on a shared zero from −1 to +1. Runs that were excluded are left out.

Where the money goes

A run of Orchestrator-workers is 3 stages. This is what each one costs, so the price of the structure is separate from the price of the work.

  • Haiku 4.5$0.171 per run · 50 runs
  • Sonnet 5.5$0.216 per run · 50 runs
  • Opus 5.5$0.459 per run · 50 runs
StageHaiku 4.5Sonnet 5.5Opus 5.5
The orchestrator$0.01811%$0.0147%$0.0327%
The workers$0.13981%$0.16777%$0.34776%
The synthesis$0.0148%$0.03516%$0.08017%
Mean cost of a ranked run, by the stage of the design that spent it. Judge calls are not counted.

Did it run as designed?

Each check is recorded with every run. A run that fails one is excluded from every ranking and counted, never scored.

The design’s own checks

  • The task was decomposedorchestrator-workers.decomposition

    The orchestrator’s reply was read through its schema and held at least one subtask.

    150/150 100%
  • The fan-out happenedorchestrator-workers.fan-out

    Every subtask got a worker, every worker ran, and the recorded subagent count matches.

    150/150 100%
  • The synthesis wrote the answerorchestrator-workers.synthesis

    The final answer came from the synthesis call, not from a worker’s report standing in for it.

    150/150 100%
  • Each call was shown the right toolsorchestrator-workers.tools

    The orchestrator and the synthesis were shown no tools and every worker the task’s full toolset.

    150/150 100%
The harness’s checksall 10 held in every run
  • No API failuresharness.api

    No model call failed at the API.

    150/150 100%
  • Builtin toolsharness.builtin-tools

    A check recorded with the run.

    150/150 100%
  • Every call endedharness.calls-ended

    Every call ended with a result, or was stopped by the run's own limits.

    150/150 100%
  • No harness tool deniedharness.harness-tools

    The sandbox guard never denied a harness tool, nor a tool the session itself offered.

    150/150 100%
  • Judge answeredharness.judge

    At least one grading call returned a usable verdict.

    18/18 100%
  • Tool servers connectedharness.mcp-connected

    Every tool server the harness gave a call was connected when the session started.

    150/150 100%
  • No foreign tool serversharness.mcp-hermetic

    No call loaded a tool server the harness did not give it.

    150/150 100%
  • No silent fallbacksharness.not-degraded

    No phase fell back on a default (an empty plan, a default winner) instead of its output.

    150/150 100%
  • Run completedharness.run-completed

    The environment was built, the architecture returned and grading finished, none of them throwing.

    150/150 100%
  • Structured replies arrivedharness.structured-output

    Every call that asked for structured output and ran to the end got it.

    150/150 100%

Harness health across every architecture →

A run of it

Orchestrator-workers on Cancel a fare the customer wrongly believes is refundable to their card with Haiku 4.5, repeat 1. It scored 0.429.

3 calls · 1m 21s · $0.068

  • tool call
  • denied by the guard
  • tool returned an error
  • shown no tools (reasoning only)
  • lost its tools (server failed)
  • failed

The whole run, message by message →

150 ranked runs of 150 recorded.

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.