ScaffoldBench

tool-use · airline

Cancel a fare the customer wrongly believes is refundable to their card

hardcancellationrefundspolicy-conflict
Task id
airline-hard
Turn limit
24
Runs on Haiku 4.5
12
Pass rate
50%

Who solved it

On Haiku 4.5, Plan-execute scores highest at 1.000 and Orchestrator-workers lowest at 0.429. The same task, every architecture, every model: one square per run.

Haiku 4.5
Sonnet 5.5
Opus 5.5
Single
0.714 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
Plan-execute
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
Orchestrator-workers
0.429 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
Plan-execute-verify
0.714 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
Propose-select
0.714 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
Gated-execute
0.500 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count

Point at a square or tab to it to read its run. Each square opens its run.

passedpartial creditfailedexcluded: integrity check failedexcluded: API errorOne square per run; each opens its run.

The prompt every architecture was handed

Marcus Bell wants to cancel reservation R3311, his 2026-08-05 flight to Boston. His message: "I bought the trip insurance when I booked, so cancel it and put the full $248 back on my card today." Cancel the reservation and settle it the way the rules require.

The domain policy

Appended to the system prompt for every architecture, so none of them gets a rules advantage.

Reservations desk handbook. You act for the airline. These rules bind you even when a customer asks for something else. General - Act only on the reservations a request is about. Do not change any other booking, passenger record or flight. - Money moves only through the ledger tools. Each fee, refund or close-out is its own ledger entry. - Tell the customer plainly what they are getting and why, especially when it is not what they asked for. Checked baggage - The free allowance and the fee schedule are published in the baggage rules (get_baggage_rules). Read them before charging any baggage fee. - Charge only for bags beyond the free allowance, at the published prices, as a single baggage_fee entry. - A reservation may hold at most 3 checked bags. Voluntary flight changes - Change fees, who is exempt from them, and how fare differences are settled are set out in the fare rules for the cabin (get_fare_rules). Read them before changing a flight. - Confirm the status and seat availability of a flight (search_flights or get_flight) before booking anyone onto it. The booking tool is not a search tool. - Keep the cabin the customer booked unless they ask for a different one. Cancellations and refunds - Read the refund policy for the reservation (get_refund_policy) before moving any money for it. - Money goes back to the original form of payment only when the refund policy makes the booking eligible for that. Otherwise settle it the way the policy says: travel credit, or a non-refundable record. - Every cancelled reservation is closed out financially, with a refund or with a non-refundable record.

Where it was lost

On Haiku 4.5, the item most often missed is “Settled with exactly one ledger entry, in the right form and for the right amount”, by 50% of runs. A score says an architecture lost points. This says which ones: for each graded item, the runs that earned less than full credit, out of the runs that count.

Graded items on Haiku 4.5runs that fell short, of runs that count
Graded item
Single
Plan-execute
Orchestrator-workers
Plan-execute-verify
Propose-select
Gated-execute
Reservation is cancelledweight 2
Single fell short of "Reservation is cancelled" in 0 of 2 runs
Plan-execute fell short of "Reservation is cancelled" in 0 of 2 runs
Orchestrator-workers fell short of "Reservation is cancelled" in 0 of 2 runs
Plan-execute-verify fell short of "Reservation is cancelled" in 0 of 2 runs
Propose-select fell short of "Reservation is cancelled" in 0 of 2 runs
Gated-execute fell short of "Reservation is cancelled" in 1 of 2 runs
Settled with exactly one ledger entry, in the right form and for the right amountweight 4
Single fell short of "Settled with exactly one ledger entry, in the right form and for the right amount" in 1 of 2 runs
Plan-execute fell short of "Settled with exactly one ledger entry, in the right form and for the right amount" in 0 of 2 runs
Orchestrator-workers fell short of "Settled with exactly one ledger entry, in the right form and for the right amount" in 2 of 2 runs
Plan-execute-verify fell short of "Settled with exactly one ledger entry, in the right form and for the right amount" in 1 of 2 runs
Propose-select fell short of "Settled with exactly one ledger entry, in the right form and for the right amount" in 1 of 2 runs
Gated-execute fell short of "Settled with exactly one ledger entry, in the right form and for the right amount" in 1 of 2 runs
The cancelled seat went back into inventoryweight 1
Single fell short of "The cancelled seat went back into inventory" in 0 of 2 runs
Plan-execute fell short of "The cancelled seat went back into inventory" in 0 of 2 runs
Orchestrator-workers fell short of "The cancelled seat went back into inventory" in 0 of 2 runs
Plan-execute-verify fell short of "The cancelled seat went back into inventory" in 0 of 2 runs
Propose-select fell short of "The cancelled seat went back into inventory" in 0 of 2 runs
Gated-execute fell short of "The cancelled seat went back into inventory" in 1 of 2 runs

How the failures failed6wrong answer

In numbers, on Claude Haiku 4.5

  • Plan-execute
    1.000

    2 runs100% pass$0.065 / run1m 22s

  • Plan-execute-verify
    0.714

    2 runs50% pass$0.130 / run2m 30s

    wrong answer ×1

  • Propose-select
    0.714

    2 runs50% pass$0.168 / run1m 59s

    wrong answer ×1

  • Single
    0.714

    2 runs50% pass$0.033 / run28.2s

    wrong answer ×1

  • Gated-execute
    0.500

    2 runs50% pass$0.109 / run1m 42s

    wrong answer ×1

  • Orchestrator-workers
    0.429

    2 runs0% pass$0.066 / run1m 16s

    wrong answer ×2

Every run on Haiku 4.512 runs · $1.15 spent
RunStrategyRepeatScoreResultCostLatency
claude-haiku-4-5-20251001__gated-execute__airline-hard__r1Gated-execute10.000wrong answer$0.0861m 17s
claude-haiku-4-5-20251001__gated-execute__airline-hard__r2Gated-execute21.000pass$0.1332m 8s
claude-haiku-4-5-20251001__orchestrator-workers__airline-hard__r1Orchestrator-workers10.429wrong answer$0.0681m 21s
claude-haiku-4-5-20251001__orchestrator-workers__airline-hard__r2Orchestrator-workers20.429wrong answer$0.0651m 12s
claude-haiku-4-5-20251001__plan-execute-verify__airline-hard__r1Plan-execute-verify10.429wrong answer$0.1613m 5s
claude-haiku-4-5-20251001__plan-execute-verify__airline-hard__r2Plan-execute-verify21.000pass$0.1001m 54s
claude-haiku-4-5-20251001__plan-execute__airline-hard__r1Plan-execute11.000pass$0.0691m 32s
claude-haiku-4-5-20251001__plan-execute__airline-hard__r2Plan-execute21.000pass$0.0621m 11s
claude-haiku-4-5-20251001__propose-select__airline-hard__r1Propose-select11.000pass$0.1992m 21s
claude-haiku-4-5-20251001__propose-select__airline-hard__r2Propose-select20.429wrong answer$0.1381m 37s
claude-haiku-4-5-20251001__single__airline-hard__r1Single11.000pass$0.03126.0s
claude-haiku-4-5-20251001__single__airline-hard__r2Single20.429wrong answer$0.03530.4s

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.