ScaffoldBench

tool-use · airline

Rebook onto the earliest available flight and settle the difference

mediumrebookingfare-rulesfees
Task id
airline-medium
Turn limit
22
Runs on Opus 5.5
12
Pass rate
100%

Who solved it

On Opus 5.5, every architecture averages 1.000: this task does not separate them. The same task, every architecture, every model: one square per run.

1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
0.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
0.786 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
0.500 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count

Point at a square or tab to it to read its run. Each square opens its run.

passedpartial creditfailedexcluded: integrity check failedexcluded: API errorOne square per run; each opens its run.

The prompt every architecture was handed

Dana Whitfield is on reservation R2087, SFO to ORD on 2026-08-03. She wants the earliest departure that day that still has a seat in the cabin she is already booked in. Move her onto it and settle whatever she owes for the change.

The domain policy

Appended to the system prompt for every architecture, so none of them gets a rules advantage.

Reservations desk handbook. You act for the airline. These rules bind you even when a customer asks for something else. General - Act only on the reservations a request is about. Do not change any other booking, passenger record or flight. - Money moves only through the ledger tools. Each fee, refund or close-out is its own ledger entry. - Tell the customer plainly what they are getting and why, especially when it is not what they asked for. Checked baggage - The free allowance and the fee schedule are published in the baggage rules (get_baggage_rules). Read them before charging any baggage fee. - Charge only for bags beyond the free allowance, at the published prices, as a single baggage_fee entry. - A reservation may hold at most 3 checked bags. Voluntary flight changes - Change fees, who is exempt from them, and how fare differences are settled are set out in the fare rules for the cabin (get_fare_rules). Read them before changing a flight. - Confirm the status and seat availability of a flight (search_flights or get_flight) before booking anyone onto it. The booking tool is not a search tool. - Keep the cabin the customer booked unless they ask for a different one. Cancellations and refunds - Read the refund policy for the reservation (get_refund_policy) before moving any money for it. - Money goes back to the original form of payment only when the refund policy makes the booking eligible for that. Otherwise settle it the way the policy says: travel credit, or a non-refundable record. - Every cancelled reservation is closed out financially, with a refund or with a non-refundable record.

Where it was lost

On Opus 5.5, no architecture fell short of any graded item. A score says an architecture lost points. This says which ones: for each graded item, the runs that earned less than full credit, out of the runs that count.

Graded items on Opus 5.5runs that fell short, of runs that count
Graded item
Single
Plan-execute
Orchestrator-workers
Plan-execute-verify
Propose-select
Gated-execute
Reservation moved to the earliest flight with an available seat, same cabinweight 3
Single fell short of "Reservation moved to the earliest flight with an available seat, same cabin" in 0 of 2 runs
Plan-execute fell short of "Reservation moved to the earliest flight with an available seat, same cabin" in 0 of 2 runs
Orchestrator-workers fell short of "Reservation moved to the earliest flight with an available seat, same cabin" in 0 of 2 runs
Plan-execute-verify fell short of "Reservation moved to the earliest flight with an available seat, same cabin" in 0 of 2 runs
Propose-select fell short of "Reservation moved to the earliest flight with an available seat, same cabin" in 0 of 2 runs
Gated-execute fell short of "Reservation moved to the earliest flight with an available seat, same cabin" in 0 of 2 runs
Exactly one fee, the fare difference for the flight actually bookedweight 3
Single fell short of "Exactly one fee, the fare difference for the flight actually booked" in 0 of 2 runs
Plan-execute fell short of "Exactly one fee, the fare difference for the flight actually booked" in 0 of 2 runs
Orchestrator-workers fell short of "Exactly one fee, the fare difference for the flight actually booked" in 0 of 2 runs
Plan-execute-verify fell short of "Exactly one fee, the fare difference for the flight actually booked" in 0 of 2 runs
Propose-select fell short of "Exactly one fee, the fare difference for the flight actually booked" in 0 of 2 runs
Gated-execute fell short of "Exactly one fee, the fare difference for the flight actually booked" in 0 of 2 runs
Exactly one economy seat moved from the old flight to the new oneweight 1
Single fell short of "Exactly one economy seat moved from the old flight to the new one" in 0 of 2 runs
Plan-execute fell short of "Exactly one economy seat moved from the old flight to the new one" in 0 of 2 runs
Orchestrator-workers fell short of "Exactly one economy seat moved from the old flight to the new one" in 0 of 2 runs
Plan-execute-verify fell short of "Exactly one economy seat moved from the old flight to the new one" in 0 of 2 runs
Propose-select fell short of "Exactly one economy seat moved from the old flight to the new one" in 0 of 2 runs
Gated-execute fell short of "Exactly one economy seat moved from the old flight to the new one" in 0 of 2 runs

In numbers, on Claude Opus 5.5

  • Gated-execute
    1.000

    2 runs100% pass$0.147 / run40.7s

  • Orchestrator-workers
    1.000

    2 runs100% pass$0.124 / run38.5s

  • Plan-execute
    1.000

    2 runs100% pass$0.111 / run32.4s

  • Plan-execute-verify
    1.000

    2 runs100% pass$0.159 / run45.0s

  • Propose-select
    1.000

    2 runs100% pass$0.343 / run57.8s

  • Single
    1.000

    2 runs100% pass$0.085 / run20.2s

Every run on Opus 5.512 runs · $1.94 spent
RunStrategyRepeatScoreResultCostLatency
claude-opus-5-5__gated-execute__airline-medium__r1Gated-execute11.000pass$0.15039.2s
claude-opus-5-5__gated-execute__airline-medium__r2Gated-execute21.000pass$0.14342.2s
claude-opus-5-5__orchestrator-workers__airline-medium__r1Orchestrator-workers11.000pass$0.13039.3s
claude-opus-5-5__orchestrator-workers__airline-medium__r2Orchestrator-workers21.000pass$0.11937.8s
claude-opus-5-5__plan-execute-verify__airline-medium__r1Plan-execute-verify11.000pass$0.16748.6s
claude-opus-5-5__plan-execute-verify__airline-medium__r2Plan-execute-verify21.000pass$0.15241.4s
claude-opus-5-5__plan-execute__airline-medium__r1Plan-execute11.000pass$0.11832.8s
claude-opus-5-5__plan-execute__airline-medium__r2Plan-execute21.000pass$0.10432.0s
claude-opus-5-5__propose-select__airline-medium__r1Propose-select11.000pass$0.34753.8s
claude-opus-5-5__propose-select__airline-medium__r2Propose-select21.000pass$0.3391m 2s
claude-opus-5-5__single__airline-medium__r1Single11.000pass$0.09819.5s
claude-opus-5-5__single__airline-medium__r2Single21.000pass$0.07221.0s

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.