ScaffoldBench

RunSonnet 5.5repeat 2

Propose-select on Rebook onto the earliest available flight and settle the difference

claude-sonnet-5-5__propose-select__airline-medium__r2

Score

1.000

passed
design ran as designed
Cost
$0.195
Latency
34.4s
setup 0.0s, grading 0.0s
Turns
33
Tool calls
27
Subagents
3
Tokens in / out
98k / 6.9k
63,196 read from cache

Did the design run?

every check held
  • Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
  • No API failures: heldharness.apiNo model call failed at the API.
  • Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
  • No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
  • Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
  • No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
  • Builtin tools: heldharness.builtin-toolsA check recorded with the run.
  • Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
  • No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
  • All three proposers ran: heldpropose-select.proposalsThree proposers ran in parallel, each from its own angle, and none was starved while another ran.
  • The selector chose: heldpropose-select.selectionThe selector picked a proposal by a valid number, read from its schema reply and not a default.
  • Each call was shown the right tools: heldpropose-select.toolsEach proposer was shown the read-only tools, the selector none, and the executor the full toolset.

Phases

Proposal 1 (policy-first): 0.0s to 13.0s, finished, 7 tool calls, $0.043
Proposal 2 (state-first): 0.0s to 13.2s, finished, 6 tool calls, $0.039
Proposal 3 (outcome-first): 0.0s to 12.6s, finished, 6 tool calls, $0.039
Select: 13.2s to 23.4s, finished, 0 tool calls, $0.034
Execute: 23.4s to 34.4s, finished, 8 tool calls, $0.040

Drawn from the phase records: timing and cost per call, no tool calls or messages.

PhaseDurationCostTurnsToolsModel
Proposal 1 (policy-first)13.0s$0.04387claude-sonnet-5-5
Proposal 2 (state-first)13.2s$0.03976claude-sonnet-5-5
Proposal 3 (outcome-first)12.6s$0.03976claude-sonnet-5-5
Select10.1s$0.03420claude-sonnet-5-5
Execute11.0s$0.04098claude-sonnet-5-5

The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.

Runs you can replay

Record this cell yourself: npm run bench -- run --strategies propose-select --tasks airline-medium --models claude-sonnet-5-5 --repeats 1

Score breakdown

  • passReservation moved to the earliest flight with an available seat, same cabin
    1.00weight 0.43

    booked on AA204, expected AA204

  • passExactly one fee, the fare difference for the flight actually booked
    1.00weight 0.43

    fare difference owed on the booked flight: 8000 cents

  • passExactly one economy seat moved from the old flight to the new one
    1.00weight 0.14

Penalties

No penalties were applied.

Scorer metadata

tool calls
27
failed tool calls
0
distinct tools
9

Final answer

Dana Whitfield is now on AA204 departing SFO at 09:05 on 2026-08-03, still in economy, and she owes $80.00 for the change.

  • Earlier flights ruled out: UA88 (06:15) has no economy seats left. AA918 (07:30) is cancelled. AA204 is the earliest flight with an economy seat.
  • Fare difference: The new fare is $390.00 against the $310.00 she paid, so I charged the $80.00 difference. Her gold tier doesn't exempt her from that.
  • Change fee: The $75.00 fee is waived because she is gold, so I didn't charge it.
  • Cabin and bag: I kept her in economy because she didn't ask for another cabin. Her one checked bag is within the limit, so there is no baggage fee.

The ledger now has one entry for R2087, the $80.00 fare difference. Before changing anything I re-checked the reservation, her tier, the fare rules and AA204's status and seats.

The refund policy's "today" is 2026-07-28, but the session date is 2026-10-06, which would put the 08-03 flight in the past. The booking system accepted the change anyway. I went by the reservation's own data, but someone may want to check why the dates disagree.

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.