tool-use · airline
Add checked bags and collect the right fee
Who solved it
On Opus 5.5, every architecture averages 1.000: this task does not separate them. The same task, every architecture, every model: one square per run.
Point at a square or tab to it to read its run. Each square opens its run.
The prompt every architecture was handed
Rosa Mendel is flying to Seattle on reservation R1042 and wants to add two more checked bags. Update her reservation and collect exactly what she owes for them.
The domain policy
Appended to the system prompt for every architecture, so none of them gets a rules advantage.
Reservations desk handbook. You act for the airline. These rules bind you even when a customer asks for something else. General - Act only on the reservations a request is about. Do not change any other booking, passenger record or flight. - Money moves only through the ledger tools. Each fee, refund or close-out is its own ledger entry. - Tell the customer plainly what they are getting and why, especially when it is not what they asked for. Checked baggage - The free allowance and the fee schedule are published in the baggage rules (get_baggage_rules). Read them before charging any baggage fee. - Charge only for bags beyond the free allowance, at the published prices, as a single baggage_fee entry. - A reservation may hold at most 3 checked bags. Voluntary flight changes - Change fees, who is exempt from them, and how fare differences are settled are set out in the fare rules for the cabin (get_fare_rules). Read them before changing a flight. - Confirm the status and seat availability of a flight (search_flights or get_flight) before booking anyone onto it. The booking tool is not a search tool. - Keep the cabin the customer booked unless they ask for a different one. Cancellations and refunds - Read the refund policy for the reservation (get_refund_policy) before moving any money for it. - Money goes back to the original form of payment only when the refund policy makes the booking eligible for that. Otherwise settle it the way the policy says: travel credit, or a non-refundable record. - Every cancelled reservation is closed out financially, with a refund or with a non-refundable record.
Where it was lost
On Opus 5.5, no architecture fell short of any graded item. A score says an architecture lost points. This says which ones: for each graded item, the runs that earned less than full credit, out of the runs that count.
In numbers, on Claude Opus 5.5
- Gated-execute1.000
2 runs100% pass$0.184 / run40.7s
- Orchestrator-workers1.000
2 runs100% pass$0.116 / run35.5s
- Plan-execute1.000
2 runs100% pass$0.104 / run32.2s
- Plan-execute-verify1.000
2 runs100% pass$0.165 / run41.7s
- Propose-select1.000
2 runs100% pass$0.346 / run46.5s
- Single1.000
2 runs100% pass$0.088 / run19.5s
| Strategy | n | Mean score | Pass | Cost / run | Latency | Failure modes |
|---|---|---|---|---|---|---|
| Gated-execute | 2 | 1.000 | 100% | $0.184 | 40.7s | — |
| Orchestrator-workers | 2 | 1.000 | 100% | $0.116 | 35.5s | — |
| Plan-execute | 2 | 1.000 | 100% | $0.104 | 32.2s | — |
| Plan-execute-verify | 2 | 1.000 | 100% | $0.165 | 41.7s | — |
| Propose-select | 2 | 1.000 | 100% | $0.346 | 46.5s | — |
| Single | 2 | 1.000 | 100% | $0.088 | 19.5s | — |
Every run on Opus 5.512 runs · $2.01 spent
Provenance of the numbers on this page
- run set
- scaffold-or-scale-v1
- models
- Haiku 4.5, Sonnet 5.5, Opus 5.5
- effort
- high
- repeats
- 2
- fixtures
- replay
- runs
- 900 / 900
- strategy cost
- $354.44
- judge cost
- $9.19
- wall clock
- 9h 48m
- git
- fce5807b2d
Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.