ScaffoldBench

tool-use · airline

Rebook onto the earliest available flight and settle the difference

mediumrebookingfare-rulesfees
Task id
airline-medium
Turn limit
22
Runs on Haiku 4.5
12
Pass rate
67%

Who solved it

On Haiku 4.5, Single, Propose-select and Gated-execute score highest at 1.000 and Plan-execute lowest at 0.000. The same task, every architecture, every model: one square per run.

1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
0.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
0.786 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
0.500 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count

Point at a square or tab to it to read its run. Each square opens its run.

passedpartial creditfailedexcluded: integrity check failedexcluded: API errorOne square per run; each opens its run.

The prompt every architecture was handed

Dana Whitfield is on reservation R2087, SFO to ORD on 2026-08-03. She wants the earliest departure that day that still has a seat in the cabin she is already booked in. Move her onto it and settle whatever she owes for the change.

The domain policy

Appended to the system prompt for every architecture, so none of them gets a rules advantage.

Reservations desk handbook. You act for the airline. These rules bind you even when a customer asks for something else. General - Act only on the reservations a request is about. Do not change any other booking, passenger record or flight. - Money moves only through the ledger tools. Each fee, refund or close-out is its own ledger entry. - Tell the customer plainly what they are getting and why, especially when it is not what they asked for. Checked baggage - The free allowance and the fee schedule are published in the baggage rules (get_baggage_rules). Read them before charging any baggage fee. - Charge only for bags beyond the free allowance, at the published prices, as a single baggage_fee entry. - A reservation may hold at most 3 checked bags. Voluntary flight changes - Change fees, who is exempt from them, and how fare differences are settled are set out in the fare rules for the cabin (get_fare_rules). Read them before changing a flight. - Confirm the status and seat availability of a flight (search_flights or get_flight) before booking anyone onto it. The booking tool is not a search tool. - Keep the cabin the customer booked unless they ask for a different one. Cancellations and refunds - Read the refund policy for the reservation (get_refund_policy) before moving any money for it. - Money goes back to the original form of payment only when the refund policy makes the booking eligible for that. Otherwise settle it the way the policy says: travel credit, or a non-refundable record. - Every cancelled reservation is closed out financially, with a refund or with a non-refundable record.

Where it was lost

On Haiku 4.5, the item most often missed is “Exactly one fee, the fare difference for the flight actually booked”, by 33% of runs. A score says an architecture lost points. This says which ones: for each graded item, the runs that earned less than full credit, out of the runs that count.

Graded items on Haiku 4.5runs that fell short, of runs that count
Graded item
Single
Plan-execute
Orchestrator-workers
Plan-execute-verify
Propose-select
Gated-execute
Reservation moved to the earliest flight with an available seat, same cabinweight 3
Single fell short of "Reservation moved to the earliest flight with an available seat, same cabin" in 0 of 2 runs
Plan-execute fell short of "Reservation moved to the earliest flight with an available seat, same cabin" in 0 of 2 runs
Orchestrator-workers fell short of "Reservation moved to the earliest flight with an available seat, same cabin" in 0 of 2 runs
Plan-execute-verify fell short of "Reservation moved to the earliest flight with an available seat, same cabin" in 0 of 2 runs
Propose-select fell short of "Reservation moved to the earliest flight with an available seat, same cabin" in 0 of 2 runs
Gated-execute fell short of "Reservation moved to the earliest flight with an available seat, same cabin" in 0 of 2 runs
Exactly one fee, the fare difference for the flight actually bookedweight 3
Single fell short of "Exactly one fee, the fare difference for the flight actually booked" in 0 of 2 runs
Plan-execute fell short of "Exactly one fee, the fare difference for the flight actually booked" in 2 of 2 runs
Orchestrator-workers fell short of "Exactly one fee, the fare difference for the flight actually booked" in 1 of 2 runs
Plan-execute-verify fell short of "Exactly one fee, the fare difference for the flight actually booked" in 1 of 2 runs
Propose-select fell short of "Exactly one fee, the fare difference for the flight actually booked" in 0 of 2 runs
Gated-execute fell short of "Exactly one fee, the fare difference for the flight actually booked" in 0 of 2 runs
Exactly one economy seat moved from the old flight to the new oneweight 1
Single fell short of "Exactly one economy seat moved from the old flight to the new one" in 0 of 2 runs
Plan-execute fell short of "Exactly one economy seat moved from the old flight to the new one" in 0 of 2 runs
Orchestrator-workers fell short of "Exactly one economy seat moved from the old flight to the new one" in 0 of 2 runs
Plan-execute-verify fell short of "Exactly one economy seat moved from the old flight to the new one" in 0 of 2 runs
Propose-select fell short of "Exactly one economy seat moved from the old flight to the new one" in 0 of 2 runs
Gated-execute fell short of "Exactly one economy seat moved from the old flight to the new one" in 0 of 2 runs
Charged a change fee to a tier the fare rules exemptA rule: breaching it multiplies the score down.
Single fell short of "Charged a change fee to a tier the fare rules exempt" in 0 of 2 runs
Plan-execute fell short of "Charged a change fee to a tier the fare rules exempt" in 2 of 2 runs
Orchestrator-workers fell short of "Charged a change fee to a tier the fare rules exempt" in 0 of 2 runs
Plan-execute-verify fell short of "Charged a change fee to a tier the fare rules exempt" in 1 of 2 runs
Propose-select fell short of "Charged a change fee to a tier the fare rules exempt" in 0 of 2 runs
Gated-execute fell short of "Charged a change fee to a tier the fare rules exempt" in 0 of 2 runs

How the failures failed3policy violation1wrong answer

In numbers, on Claude Haiku 4.5

  • Gated-execute
    1.000

    2 runs100% pass$0.087 / run1m 10s

  • Propose-select
    1.000

    2 runs100% pass$0.137 / run1m 32s

  • Single
    1.000

    2 runs100% pass$0.029 / run19.4s

  • Orchestrator-workers
    0.786

    2 runs50% pass$0.088 / run1m 27s

    wrong answer ×1

  • Plan-execute-verify
    0.500

    2 runs50% pass$0.112 / run1m 50s

    policy violation ×1

  • Plan-execute
    0.000

    2 runs0% pass$0.041 / run42.9s

    policy violation ×2

Every run on Haiku 4.512 runs · $0.987 spent
RunStrategyRepeatScoreResultCostLatency
claude-haiku-4-5-20251001__gated-execute__airline-medium__r1Gated-execute11.000pass$0.0861m 11s
claude-haiku-4-5-20251001__gated-execute__airline-medium__r2Gated-execute21.000pass$0.0881m 9s
claude-haiku-4-5-20251001__orchestrator-workers__airline-medium__r1Orchestrator-workers11.000pass$0.05456.2s
claude-haiku-4-5-20251001__orchestrator-workers__airline-medium__r2Orchestrator-workers20.571wrong answer$0.1221m 58s
claude-haiku-4-5-20251001__plan-execute-verify__airline-medium__r1Plan-execute-verify11.000pass$0.0771m 19s
claude-haiku-4-5-20251001__plan-execute-verify__airline-medium__r2Plan-execute-verify20.000policy violation$0.1472m 21s
claude-haiku-4-5-20251001__plan-execute__airline-medium__r1Plan-execute10.000policy violation$0.04144.9s
claude-haiku-4-5-20251001__plan-execute__airline-medium__r2Plan-execute20.000policy violation$0.04041.0s
claude-haiku-4-5-20251001__propose-select__airline-medium__r1Propose-select11.000pass$0.1361m 37s
claude-haiku-4-5-20251001__propose-select__airline-medium__r2Propose-select21.000pass$0.1391m 27s
claude-haiku-4-5-20251001__single__airline-medium__r1Single11.000pass$0.02818.2s
claude-haiku-4-5-20251001__single__airline-medium__r2Single21.000pass$0.03020.7s

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.