ScaffoldBench

tool-use · airline

Re-protect a cancelled flight when seats are scarce

expertirregular-operationsscarce-resourcepriorityconnectionslong-horizon
Task id
airline-irrops-reprotect
Turn limit
120
Runs on Opus 5.5
12
Pass rate
100%

Pre-registered hypothesis

Hypothesis

single >= propose-select > orchestrator-workers. Thirteen reservations compete for one or two seats per cabin, and whoever books first takes the seat: an upgrade granted to a gold passenger removes the seat a later passenger would have had. Workers acting in parallel on slices of the manifest each grab the earliest seat they see, inverting the tier-then-date priority; read-only proposers see the whole board before any seat is taken, and a single agent can sort once and allocate in order. A verifier (plan-execute-verify) can detect an inversion but must unwind bookings to repair it.

Written when the task was written, before any run. It is a prediction in words, so read it against what happened rather than trusting a verdict drawn for you.

What happened on Opus 5.5

  • Highest mean score: Single 1.000
  • Lowest: Gated-execute 1.000

Means over the runs that count; intervals and verdicts are on the results page.

Who solved it

On Opus 5.5, every architecture averages 1.000: this task does not separate them. The same task, every architecture, every model: one square per run.

0.692 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
0.702 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
0.615 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
0.327 mean score of the runs that count
0.885 mean score of the runs that count
1.000 mean score of the runs that count
0.750 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
0.256 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count

Point at a square or tab to it to read its run. Each square opens its run.

passedpartial creditfailedexcluded: integrity check failedexcluded: API errorOne square per run; each opens its run.

The prompt every architecture was handed

Operations has cancelled AA918 (SFO to ORD, 2026-08-03, departing 07:30). Re-protect every passenger booked on it under the irregular-operations rules, issue whatever they are owed, and let each of them know where they are now booked.

The domain policy

Appended to the system prompt for every architecture, so none of them gets a rules advantage.

Reservations desk handbook. You act for the airline. These rules bind you even when a customer asks for something else. General - Act only on the reservations a request is about. Do not change any other booking, passenger record or flight. - Money moves only through the ledger tools. Each fee, refund or close-out is its own ledger entry. - Tell the customer plainly what they are getting and why, especially when it is not what they asked for. Checked baggage - The free allowance and the fee schedule are published in the baggage rules (get_baggage_rules). Read them before charging any baggage fee. - Charge only for bags beyond the free allowance, at the published prices, as a single baggage_fee entry. - A reservation may hold at most 3 checked bags. Voluntary flight changes - Change fees, who is exempt from them, and how fare differences are settled are set out in the fare rules for the cabin (get_fare_rules). Read them before changing a flight. - Confirm the status and seat availability of a flight (search_flights or get_flight) before booking anyone onto it. The booking tool is not a search tool. - Keep the cabin the customer booked unless they ask for a different one. Cancellations and refunds - Read the refund policy for the reservation (get_refund_policy) before moving any money for it. - Money goes back to the original form of payment only when the refund policy makes the booking eligible for that. Otherwise settle it the way the policy says: travel credit, or a non-refundable record. - Every cancelled reservation is closed out financially, with a refund or with a non-refundable record. Request queue - Customer requests arrive in the service queue (list_requests, get_request). Work the requests you are asked to work and leave the rest for the teams that own them. - Resolve each request you complete (resolve_request) with a short note to the customer saying what was done and what they are getting. Payments - Money can only be sent to a payment instrument that is active. When the instrument a booking was paid with is no longer active, issue the amount as travel credit instead and tell the customer why. Operational bulletins - Operations publishes dated bulletins (list_bulletins, get_bulletin). A bulletin in effect on the day you act overrides this handbook and the published rule tables within its stated scope. Outside its dates or its scope it has no effect, whoever cites it. Irregular operations (flights cancelled by the airline) - Passengers on a flight the airline cancelled are re-protected free of charge: no change fee and no fare difference, whatever the fare rules say about voluntary changes. - When there are not enough seats for everyone, work through the passengers in priority order: frequent-flyer tier (platinum, gold, silver, none), then earliest booking date. Each takes the available itinerary that reaches their final destination soonest. - Re-protect each passenger in the cabin they paid for, never lower. The one exception runs upwards: a gold or platinum passenger travelling alone on a single flight, whose own cabin is full on a flight that would get them to their destination sooner than any itinerary still open in their own cabin, is seated one cabin higher on that flight. Nobody else is moved up a cabin, and nobody is moved up more than one. - Reservations booked to travel together are re-protected together, on the same flights and in their own cabin, at the priority of their highest-priority member. They are never split and never upgraded. - A connection is valid only when the onward flight departs at least 60 minutes after the previous flight arrives; an onward flight the next morning is a valid connection. When a new first leg breaks a connection, rebook the onward leg too. - Compensation depends on how much later than originally scheduled the passenger reaches their final destination: under 3 hours, none; from 3 hours up to but not including 6 hours, a 15000-cent compensation voucher; 6 hours or more, a 30000-cent compensation voucher. A compensation voucher never exceeds the fare paid for the reservation. - A passenger who reaches their final destination on a later date than originally scheduled also receives one hotel voucher of 18000 cents. - Tell each re-protected passenger their new itinerary once, after it is booked. Fee audits - An audited fee is judged by the rules in force when it was charged, applied to the passenger as they were on that date. - Refund an overcharge to the original payment method for exactly the amount overcharged (refund_fee). Never re-bill an undercharge. - A reservation pays the baggage fee for its bags once. A second baggage_fee for the same bags is refunded in full, and the first is audited on its own merits. - Record the outcome on every entry you audit (annotate_ledger_entry): correct, overcharged or undercharged.

The traps it sets

A trap is a specific way to go wrong, declared before any run and tied to the assertions or rules that register it. A trap fired in a run when one of those failed. On Opus 5.5:

Re-protect a cancelled flight when seats are scarce10 traps · runs that fell in, of runs ranked
Trap
Single
Plan-execute
Orchestrator-workers
Plan-execute-verify
Propose-select
Gated-execute
priority-inversionAllocates scarce seats in manifest or booking-date order (or in parallel) instead of tier, then booking date.
Single fell into priority-inversion in 0 of 2 runs
Plan-execute fell into priority-inversion in 0 of 2 runs
Orchestrator-workers fell into priority-inversion in 0 of 2 runs
Plan-execute-verify fell into priority-inversion in 0 of 2 runs
Propose-select fell into priority-inversion in 0 of 2 runs
Gated-execute fell into priority-inversion in 0 of 2 runs
missed-upgradeLeaves a platinum passenger in a later economy seat when their economy cabin is full on an earlier flight with room one cabin up.
Single fell into missed-upgrade in 0 of 2 runs
Plan-execute fell into missed-upgrade in 0 of 2 runs
Orchestrator-workers fell into missed-upgrade in 0 of 2 runs
Plan-execute-verify fell into missed-upgrade in 0 of 2 runs
Propose-select fell into missed-upgrade in 0 of 2 runs
Gated-execute fell into missed-upgrade in 0 of 2 runs
upgrade-overreachMoves a gold passenger up two cabins, or upgrades someone the rule does not cover, taking a seat from a passenger further down.
Single fell into upgrade-overreach in 0 of 2 runs
Plan-execute fell into upgrade-overreach in 0 of 2 runs
Orchestrator-workers fell into upgrade-overreach in 0 of 2 runs
Plan-execute-verify fell into upgrade-overreach in 0 of 2 runs
Propose-select fell into upgrade-overreach in 0 of 2 runs
Gated-execute fell into upgrade-overreach in 0 of 2 runs
split-partySplits the two reservations booked together across flights, or upgrades one of them.
Single fell into split-party in 0 of 2 runs
Plan-execute fell into split-party in 0 of 2 runs
Orchestrator-workers fell into split-party in 0 of 2 runs
Plan-execute-verify fell into split-party in 0 of 2 runs
Propose-select fell into split-party in 0 of 2 runs
Gated-execute fell into split-party in 0 of 2 runs
overnight-connectionLeaves a connecting passenger's onward flight in place after it has become unreachable, instead of rebooking it to the next morning.
Single fell into overnight-connection in 0 of 2 runs
Plan-execute fell into overnight-connection in 0 of 2 runs
Orchestrator-workers fell into overnight-connection in 0 of 2 runs
Plan-execute-verify fell into overnight-connection in 0 of 2 runs
Propose-select fell into overnight-connection in 0 of 2 runs
Gated-execute fell into overnight-connection in 0 of 2 runs
delay-at-the-hubMeasures a connecting passenger's delay at Chicago rather than at the final destination, missing the hotel and the higher band.
Single fell into delay-at-the-hub in 0 of 2 runs
Plan-execute fell into delay-at-the-hub in 0 of 2 runs
Orchestrator-workers fell into delay-at-the-hub in 0 of 2 runs
Plan-execute-verify fell into delay-at-the-hub in 0 of 2 runs
Propose-select fell into delay-at-the-hub in 0 of 2 runs
Gated-execute fell into delay-at-the-hub in 0 of 2 runs
fare-capIssues a compensation voucher larger than the fare the passenger paid.
Single fell into fare-cap in 0 of 2 runs
Plan-execute fell into fare-cap in 0 of 2 runs
Orchestrator-workers fell into fare-cap in 0 of 2 runs
Plan-execute-verify fell into fare-cap in 0 of 2 runs
Propose-select fell into fare-cap in 0 of 2 runs
Gated-execute fell into fare-cap in 0 of 2 runs
charges-involuntary-changeApplies voluntary change fees or fare differences to an airline-cancelled flight.
Single fell into charges-involuntary-change in 0 of 2 runs
Plan-execute fell into charges-involuntary-change in 0 of 2 runs
Orchestrator-workers fell into charges-involuntary-change in 0 of 2 runs
Plan-execute-verify fell into charges-involuntary-change in 0 of 2 runs
Propose-select fell into charges-involuntary-change in 0 of 2 runs
Gated-execute fell into charges-involuntary-change in 0 of 2 runs
refunds-reprotected-fareRefunds the fare, in cash or as credit, of a passenger it has also re-protected, as though the cancellation were theirs.
Single fell into refunds-reprotected-fare in 0 of 2 runs
Plan-execute fell into refunds-reprotected-fare in 0 of 2 runs
Orchestrator-workers fell into refunds-reprotected-fare in 0 of 2 runs
Plan-execute-verify fell into refunds-reprotected-fare in 0 of 2 runs
Propose-select fell into refunds-reprotected-fare in 0 of 2 runs
Gated-execute fell into refunds-reprotected-fare in 0 of 2 runs
downgrade-to-fitSqueezes a passenger into a lower cabin to get them out sooner.
Single fell into downgrade-to-fit in 0 of 2 runs
Plan-execute fell into downgrade-to-fit in 0 of 2 runs
Orchestrator-workers fell into downgrade-to-fit in 0 of 2 runs
Plan-execute-verify fell into downgrade-to-fit in 0 of 2 runs
Propose-select fell into downgrade-to-fit in 0 of 2 runs
Gated-execute fell into downgrade-to-fit in 0 of 2 runs

Where it was lost

On Opus 5.5, no architecture fell short of any graded item. A score says an architecture lost points. This says which ones: for each graded item, the runs that earned less than full credit, out of the runs that count.

Graded items on Opus 5.5runs that fell short, of runs that count
Graded item
Single
Plan-execute
Orchestrator-workers
Plan-execute-verify
Propose-select
Gated-execute
R6101 re-protected on the prescribed itinerary and cabinweight 2
Single fell short of "R6101 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Plan-execute fell short of "R6101 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Orchestrator-workers fell short of "R6101 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Plan-execute-verify fell short of "R6101 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Propose-select fell short of "R6101 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Gated-execute fell short of "R6101 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
R6101 got exactly the vouchers its new itinerary earnsweight 1
Single fell short of "R6101 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Plan-execute fell short of "R6101 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Orchestrator-workers fell short of "R6101 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Plan-execute-verify fell short of "R6101 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Propose-select fell short of "R6101 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Gated-execute fell short of "R6101 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
R6101: passenger told their new itinerary exactly onceweight 1
Single fell short of "R6101: passenger told their new itinerary exactly once" in 0 of 2 runs
Plan-execute fell short of "R6101: passenger told their new itinerary exactly once" in 0 of 2 runs
Orchestrator-workers fell short of "R6101: passenger told their new itinerary exactly once" in 0 of 2 runs
Plan-execute-verify fell short of "R6101: passenger told their new itinerary exactly once" in 0 of 2 runs
Propose-select fell short of "R6101: passenger told their new itinerary exactly once" in 0 of 2 runs
Gated-execute fell short of "R6101: passenger told their new itinerary exactly once" in 0 of 2 runs
R6102 re-protected on the prescribed itinerary and cabinweight 2
Single fell short of "R6102 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Plan-execute fell short of "R6102 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Orchestrator-workers fell short of "R6102 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Plan-execute-verify fell short of "R6102 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Propose-select fell short of "R6102 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Gated-execute fell short of "R6102 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
R6102 got exactly the vouchers its new itinerary earnsweight 1
Single fell short of "R6102 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Plan-execute fell short of "R6102 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Orchestrator-workers fell short of "R6102 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Plan-execute-verify fell short of "R6102 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Propose-select fell short of "R6102 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Gated-execute fell short of "R6102 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
R6102: passenger told their new itinerary exactly onceweight 1
Single fell short of "R6102: passenger told their new itinerary exactly once" in 0 of 2 runs
Plan-execute fell short of "R6102: passenger told their new itinerary exactly once" in 0 of 2 runs
Orchestrator-workers fell short of "R6102: passenger told their new itinerary exactly once" in 0 of 2 runs
Plan-execute-verify fell short of "R6102: passenger told their new itinerary exactly once" in 0 of 2 runs
Propose-select fell short of "R6102: passenger told their new itinerary exactly once" in 0 of 2 runs
Gated-execute fell short of "R6102: passenger told their new itinerary exactly once" in 0 of 2 runs
R6103 re-protected on the prescribed itinerary and cabinweight 2
Single fell short of "R6103 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Plan-execute fell short of "R6103 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Orchestrator-workers fell short of "R6103 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Plan-execute-verify fell short of "R6103 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Propose-select fell short of "R6103 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Gated-execute fell short of "R6103 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
R6103 got exactly the vouchers its new itinerary earnsweight 1
Single fell short of "R6103 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Plan-execute fell short of "R6103 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Orchestrator-workers fell short of "R6103 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Plan-execute-verify fell short of "R6103 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Propose-select fell short of "R6103 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Gated-execute fell short of "R6103 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
R6103: passenger told their new itinerary exactly onceweight 1
Single fell short of "R6103: passenger told their new itinerary exactly once" in 0 of 2 runs
Plan-execute fell short of "R6103: passenger told their new itinerary exactly once" in 0 of 2 runs
Orchestrator-workers fell short of "R6103: passenger told their new itinerary exactly once" in 0 of 2 runs
Plan-execute-verify fell short of "R6103: passenger told their new itinerary exactly once" in 0 of 2 runs
Propose-select fell short of "R6103: passenger told their new itinerary exactly once" in 0 of 2 runs
Gated-execute fell short of "R6103: passenger told their new itinerary exactly once" in 0 of 2 runs
R6104 re-protected on the prescribed itinerary and cabinweight 2
Single fell short of "R6104 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Plan-execute fell short of "R6104 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Orchestrator-workers fell short of "R6104 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Plan-execute-verify fell short of "R6104 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Propose-select fell short of "R6104 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Gated-execute fell short of "R6104 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
R6104 got exactly the vouchers its new itinerary earnsweight 1
Single fell short of "R6104 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Plan-execute fell short of "R6104 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Orchestrator-workers fell short of "R6104 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Plan-execute-verify fell short of "R6104 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Propose-select fell short of "R6104 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Gated-execute fell short of "R6104 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
R6104: passenger told their new itinerary exactly onceweight 1
Single fell short of "R6104: passenger told their new itinerary exactly once" in 0 of 2 runs
Plan-execute fell short of "R6104: passenger told their new itinerary exactly once" in 0 of 2 runs
Orchestrator-workers fell short of "R6104: passenger told their new itinerary exactly once" in 0 of 2 runs
Plan-execute-verify fell short of "R6104: passenger told their new itinerary exactly once" in 0 of 2 runs
Propose-select fell short of "R6104: passenger told their new itinerary exactly once" in 0 of 2 runs
Gated-execute fell short of "R6104: passenger told their new itinerary exactly once" in 0 of 2 runs
R6105 re-protected on the prescribed itinerary and cabinweight 2
Single fell short of "R6105 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Plan-execute fell short of "R6105 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Orchestrator-workers fell short of "R6105 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Plan-execute-verify fell short of "R6105 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Propose-select fell short of "R6105 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Gated-execute fell short of "R6105 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
R6105 got exactly the vouchers its new itinerary earnsweight 1
Single fell short of "R6105 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Plan-execute fell short of "R6105 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Orchestrator-workers fell short of "R6105 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Plan-execute-verify fell short of "R6105 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Propose-select fell short of "R6105 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Gated-execute fell short of "R6105 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
R6105: passenger told their new itinerary exactly onceweight 1
Single fell short of "R6105: passenger told their new itinerary exactly once" in 0 of 2 runs
Plan-execute fell short of "R6105: passenger told their new itinerary exactly once" in 0 of 2 runs
Orchestrator-workers fell short of "R6105: passenger told their new itinerary exactly once" in 0 of 2 runs
Plan-execute-verify fell short of "R6105: passenger told their new itinerary exactly once" in 0 of 2 runs
Propose-select fell short of "R6105: passenger told their new itinerary exactly once" in 0 of 2 runs
Gated-execute fell short of "R6105: passenger told their new itinerary exactly once" in 0 of 2 runs
R6106 re-protected on the prescribed itinerary and cabinweight 2
Single fell short of "R6106 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Plan-execute fell short of "R6106 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Orchestrator-workers fell short of "R6106 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Plan-execute-verify fell short of "R6106 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Propose-select fell short of "R6106 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Gated-execute fell short of "R6106 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
R6106 got exactly the vouchers its new itinerary earnsweight 1
Single fell short of "R6106 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Plan-execute fell short of "R6106 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Orchestrator-workers fell short of "R6106 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Plan-execute-verify fell short of "R6106 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Propose-select fell short of "R6106 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Gated-execute fell short of "R6106 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
R6106: passenger told their new itinerary exactly onceweight 1
Single fell short of "R6106: passenger told their new itinerary exactly once" in 0 of 2 runs
Plan-execute fell short of "R6106: passenger told their new itinerary exactly once" in 0 of 2 runs
Orchestrator-workers fell short of "R6106: passenger told their new itinerary exactly once" in 0 of 2 runs
Plan-execute-verify fell short of "R6106: passenger told their new itinerary exactly once" in 0 of 2 runs
Propose-select fell short of "R6106: passenger told their new itinerary exactly once" in 0 of 2 runs
Gated-execute fell short of "R6106: passenger told their new itinerary exactly once" in 0 of 2 runs
R6107 re-protected on the prescribed itinerary and cabinweight 2
Single fell short of "R6107 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Plan-execute fell short of "R6107 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Orchestrator-workers fell short of "R6107 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Plan-execute-verify fell short of "R6107 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Propose-select fell short of "R6107 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Gated-execute fell short of "R6107 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
R6107 got exactly the vouchers its new itinerary earnsweight 1
Single fell short of "R6107 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Plan-execute fell short of "R6107 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Orchestrator-workers fell short of "R6107 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Plan-execute-verify fell short of "R6107 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Propose-select fell short of "R6107 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Gated-execute fell short of "R6107 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
R6107: passenger told their new itinerary exactly onceweight 1
Single fell short of "R6107: passenger told their new itinerary exactly once" in 0 of 2 runs
Plan-execute fell short of "R6107: passenger told their new itinerary exactly once" in 0 of 2 runs
Orchestrator-workers fell short of "R6107: passenger told their new itinerary exactly once" in 0 of 2 runs
Plan-execute-verify fell short of "R6107: passenger told their new itinerary exactly once" in 0 of 2 runs
Propose-select fell short of "R6107: passenger told their new itinerary exactly once" in 0 of 2 runs
Gated-execute fell short of "R6107: passenger told their new itinerary exactly once" in 0 of 2 runs
R6108 re-protected on the prescribed itinerary and cabinweight 2
Single fell short of "R6108 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Plan-execute fell short of "R6108 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Orchestrator-workers fell short of "R6108 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Plan-execute-verify fell short of "R6108 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Propose-select fell short of "R6108 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Gated-execute fell short of "R6108 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
R6108 got exactly the vouchers its new itinerary earnsweight 1
Single fell short of "R6108 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Plan-execute fell short of "R6108 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Orchestrator-workers fell short of "R6108 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Plan-execute-verify fell short of "R6108 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Propose-select fell short of "R6108 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Gated-execute fell short of "R6108 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
R6108: passenger told their new itinerary exactly onceweight 1
Single fell short of "R6108: passenger told their new itinerary exactly once" in 0 of 2 runs
Plan-execute fell short of "R6108: passenger told their new itinerary exactly once" in 0 of 2 runs
Orchestrator-workers fell short of "R6108: passenger told their new itinerary exactly once" in 0 of 2 runs
Plan-execute-verify fell short of "R6108: passenger told their new itinerary exactly once" in 0 of 2 runs
Propose-select fell short of "R6108: passenger told their new itinerary exactly once" in 0 of 2 runs
Gated-execute fell short of "R6108: passenger told their new itinerary exactly once" in 0 of 2 runs
R6109 re-protected on the prescribed itinerary and cabinweight 2
Single fell short of "R6109 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Plan-execute fell short of "R6109 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Orchestrator-workers fell short of "R6109 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Plan-execute-verify fell short of "R6109 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Propose-select fell short of "R6109 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Gated-execute fell short of "R6109 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
R6109 got exactly the vouchers its new itinerary earnsweight 1
Single fell short of "R6109 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Plan-execute fell short of "R6109 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Orchestrator-workers fell short of "R6109 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Plan-execute-verify fell short of "R6109 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Propose-select fell short of "R6109 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Gated-execute fell short of "R6109 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
R6109: passenger told their new itinerary exactly onceweight 1
Single fell short of "R6109: passenger told their new itinerary exactly once" in 0 of 2 runs
Plan-execute fell short of "R6109: passenger told their new itinerary exactly once" in 0 of 2 runs
Orchestrator-workers fell short of "R6109: passenger told their new itinerary exactly once" in 0 of 2 runs
Plan-execute-verify fell short of "R6109: passenger told their new itinerary exactly once" in 0 of 2 runs
Propose-select fell short of "R6109: passenger told their new itinerary exactly once" in 0 of 2 runs
Gated-execute fell short of "R6109: passenger told their new itinerary exactly once" in 0 of 2 runs
R6110 re-protected on the prescribed itinerary and cabinweight 2
Single fell short of "R6110 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Plan-execute fell short of "R6110 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Orchestrator-workers fell short of "R6110 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Plan-execute-verify fell short of "R6110 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Propose-select fell short of "R6110 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Gated-execute fell short of "R6110 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
R6110 got exactly the vouchers its new itinerary earnsweight 1
Single fell short of "R6110 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Plan-execute fell short of "R6110 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Orchestrator-workers fell short of "R6110 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Plan-execute-verify fell short of "R6110 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Propose-select fell short of "R6110 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Gated-execute fell short of "R6110 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
R6110: passenger told their new itinerary exactly onceweight 1
Single fell short of "R6110: passenger told their new itinerary exactly once" in 0 of 2 runs
Plan-execute fell short of "R6110: passenger told their new itinerary exactly once" in 0 of 2 runs
Orchestrator-workers fell short of "R6110: passenger told their new itinerary exactly once" in 0 of 2 runs
Plan-execute-verify fell short of "R6110: passenger told their new itinerary exactly once" in 0 of 2 runs
Propose-select fell short of "R6110: passenger told their new itinerary exactly once" in 0 of 2 runs
Gated-execute fell short of "R6110: passenger told their new itinerary exactly once" in 0 of 2 runs
R6111 re-protected on the prescribed itinerary and cabinweight 2
Single fell short of "R6111 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Plan-execute fell short of "R6111 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Orchestrator-workers fell short of "R6111 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Plan-execute-verify fell short of "R6111 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Propose-select fell short of "R6111 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Gated-execute fell short of "R6111 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
R6111 got exactly the vouchers its new itinerary earnsweight 1
Single fell short of "R6111 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Plan-execute fell short of "R6111 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Orchestrator-workers fell short of "R6111 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Plan-execute-verify fell short of "R6111 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Propose-select fell short of "R6111 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Gated-execute fell short of "R6111 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
R6111: passenger told their new itinerary exactly onceweight 1
Single fell short of "R6111: passenger told their new itinerary exactly once" in 0 of 2 runs
Plan-execute fell short of "R6111: passenger told their new itinerary exactly once" in 0 of 2 runs
Orchestrator-workers fell short of "R6111: passenger told their new itinerary exactly once" in 0 of 2 runs
Plan-execute-verify fell short of "R6111: passenger told their new itinerary exactly once" in 0 of 2 runs
Propose-select fell short of "R6111: passenger told their new itinerary exactly once" in 0 of 2 runs
Gated-execute fell short of "R6111: passenger told their new itinerary exactly once" in 0 of 2 runs
R6112 re-protected on the prescribed itinerary and cabinweight 2
Single fell short of "R6112 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Plan-execute fell short of "R6112 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Orchestrator-workers fell short of "R6112 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Plan-execute-verify fell short of "R6112 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Propose-select fell short of "R6112 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Gated-execute fell short of "R6112 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
R6112 got exactly the vouchers its new itinerary earnsweight 1
Single fell short of "R6112 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Plan-execute fell short of "R6112 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Orchestrator-workers fell short of "R6112 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Plan-execute-verify fell short of "R6112 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Propose-select fell short of "R6112 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Gated-execute fell short of "R6112 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
R6112: passenger told their new itinerary exactly onceweight 1
Single fell short of "R6112: passenger told their new itinerary exactly once" in 0 of 2 runs
Plan-execute fell short of "R6112: passenger told their new itinerary exactly once" in 0 of 2 runs
Orchestrator-workers fell short of "R6112: passenger told their new itinerary exactly once" in 0 of 2 runs
Plan-execute-verify fell short of "R6112: passenger told their new itinerary exactly once" in 0 of 2 runs
Propose-select fell short of "R6112: passenger told their new itinerary exactly once" in 0 of 2 runs
Gated-execute fell short of "R6112: passenger told their new itinerary exactly once" in 0 of 2 runs
R6113 re-protected on the prescribed itinerary and cabinweight 2
Single fell short of "R6113 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Plan-execute fell short of "R6113 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Orchestrator-workers fell short of "R6113 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Plan-execute-verify fell short of "R6113 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Propose-select fell short of "R6113 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
Gated-execute fell short of "R6113 re-protected on the prescribed itinerary and cabin" in 0 of 2 runs
R6113 got exactly the vouchers its new itinerary earnsweight 1
Single fell short of "R6113 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Plan-execute fell short of "R6113 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Orchestrator-workers fell short of "R6113 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Plan-execute-verify fell short of "R6113 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Propose-select fell short of "R6113 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
Gated-execute fell short of "R6113 got exactly the vouchers its new itinerary earns" in 0 of 2 runs
R6113: passenger told their new itinerary exactly onceweight 1
Single fell short of "R6113: passenger told their new itinerary exactly once" in 0 of 2 runs
Plan-execute fell short of "R6113: passenger told their new itinerary exactly once" in 0 of 2 runs
Orchestrator-workers fell short of "R6113: passenger told their new itinerary exactly once" in 0 of 2 runs
Plan-execute-verify fell short of "R6113: passenger told their new itinerary exactly once" in 0 of 2 runs
Propose-select fell short of "R6113: passenger told their new itinerary exactly once" in 0 of 2 runs
Gated-execute fell short of "R6113: passenger told their new itinerary exactly once" in 0 of 2 runs

In numbers, on Claude Opus 5.5

  • Gated-execute
    1.000

    2 runs100% pass$3.33 / run11m 27s

  • Orchestrator-workers
    1.000

    2 runs100% pass$0.738 / run2m 54s

  • Plan-execute
    1.000

    2 runs100% pass$0.659 / run2m 43s

  • Plan-execute-verify
    1.000

    2 runs100% pass$1.01 / run3m 47s

  • Propose-select
    1.000

    2 runs100% pass$1.81 / run3m 50s

  • Single
    1.000

    2 runs100% pass$0.603 / run2m 4s

Every run on Opus 5.512 runs · $16.29 spent
RunStrategyRepeatScoreResultCostLatency
claude-opus-5-5__gated-execute__airline-irrops-reprotect__r1Gated-execute11.000pass$3.3011m 11s
claude-opus-5-5__gated-execute__airline-irrops-reprotect__r2Gated-execute21.000pass$3.3511m 44s
claude-opus-5-5__orchestrator-workers__airline-irrops-reprotect__r1Orchestrator-workers11.000pass$0.7022m 44s
claude-opus-5-5__orchestrator-workers__airline-irrops-reprotect__r2Orchestrator-workers21.000pass$0.7743m 4s
claude-opus-5-5__plan-execute-verify__airline-irrops-reprotect__r1Plan-execute-verify11.000pass$1.013m 43s
claude-opus-5-5__plan-execute-verify__airline-irrops-reprotect__r2Plan-execute-verify21.000pass$1.013m 51s
claude-opus-5-5__plan-execute__airline-irrops-reprotect__r1Plan-execute11.000pass$0.7232m 47s
claude-opus-5-5__plan-execute__airline-irrops-reprotect__r2Plan-execute21.000pass$0.5952m 39s
claude-opus-5-5__propose-select__airline-irrops-reprotect__r1Propose-select11.000pass$1.793m 53s
claude-opus-5-5__propose-select__airline-irrops-reprotect__r2Propose-select21.000pass$1.833m 47s
claude-opus-5-5__single__airline-irrops-reprotect__r1Single11.000pass$0.6002m 4s
claude-opus-5-5__single__airline-irrops-reprotect__r2Single21.000pass$0.6062m 4s

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.