ScaffoldBench

tool-use · airline

Settle change requests under dated bulletins

expertbulletinspolicy-overriderules-via-toolsbatch
Task id
airline-waiver-desk
Turn limit
120
Runs on Haiku 4.5
12
Pass rate
0%

Pre-registered hypothesis

Hypothesis

plan-execute-verify > single > plan-execute. Every settlement turns on dated bulletins reachable only through tools, two of which can cover the same move, and customers cite waivers that do not cover them; one booking has already used its single waiver move, which only its history shows. A state-grounded verifier that re-reads the bulletins and the booking against the ledger can catch a misapplied waiver and charge the remainder; a plan written before any bulletin is read inherits the customers' framing. The requests are independent, so orchestrator-workers should match single on score at lower latency, and two arrive mid-run, which only an agent that re-reads the queue will see.

Written when the task was written, before any run. It is a prediction in words, so read it against what happened rather than trusting a verdict drawn for you.

What happened on Haiku 4.5

  • Highest mean score: Plan-execute-verify 0.792
  • A single agent: 0.662
  • Lowest: Orchestrator-workers 0.000

Means over the runs that count; intervals and verdicts are on the results page.

Who solved it

On Haiku 4.5, Plan-execute-verify scores highest at 0.792 and Orchestrator-workers lowest at 0.000. The same task, every architecture, every model: one square per run.

0.662 mean score of the runs that count
0.818 mean score of the runs that count
0.818 mean score of the runs that count
0.688 mean score of the runs that count
0.766 mean score of the runs that count
0.818 mean score of the runs that count
0.000 mean score of the runs that count
0.818 mean score of the runs that count
0.818 mean score of the runs that count
0.792 mean score of the runs that count
0.818 mean score of the runs that count
0.818 mean score of the runs that count
0.727 mean score of the runs that count
0.818 mean score of the runs that count
0.818 mean score of the runs that count
0.305 mean score of the runs that count
0.818 mean score of the runs that count
0.818 mean score of the runs that count

Point at a square or tab to it to read its run. Each square opens its run.

passedpartial creditfailedexcluded: integrity check failedexcluded: API errorOne square per run; each opens its run.

The prompt every architecture was handed

The service queue has open flight-change requests from yesterday. Work all of them: make each change the customer asked for and settle it correctly.

The domain policy

Appended to the system prompt for every architecture, so none of them gets a rules advantage.

Reservations desk handbook. You act for the airline. These rules bind you even when a customer asks for something else. General - Act only on the reservations a request is about. Do not change any other booking, passenger record or flight. - Money moves only through the ledger tools. Each fee, refund or close-out is its own ledger entry. - Tell the customer plainly what they are getting and why, especially when it is not what they asked for. Checked baggage - The free allowance and the fee schedule are published in the baggage rules (get_baggage_rules). Read them before charging any baggage fee. - Charge only for bags beyond the free allowance, at the published prices, as a single baggage_fee entry. - A reservation may hold at most 3 checked bags. Voluntary flight changes - Change fees, who is exempt from them, and how fare differences are settled are set out in the fare rules for the cabin (get_fare_rules). Read them before changing a flight. - Confirm the status and seat availability of a flight (search_flights or get_flight) before booking anyone onto it. The booking tool is not a search tool. - Keep the cabin the customer booked unless they ask for a different one. Cancellations and refunds - Read the refund policy for the reservation (get_refund_policy) before moving any money for it. - Money goes back to the original form of payment only when the refund policy makes the booking eligible for that. Otherwise settle it the way the policy says: travel credit, or a non-refundable record. - Every cancelled reservation is closed out financially, with a refund or with a non-refundable record. Request queue - Customer requests arrive in the service queue (list_requests, get_request). Work the requests you are asked to work and leave the rest for the teams that own them. - Resolve each request you complete (resolve_request) with a short note to the customer saying what was done and what they are getting. Payments - Money can only be sent to a payment instrument that is active. When the instrument a booking was paid with is no longer active, issue the amount as travel credit instead and tell the customer why. Operational bulletins - Operations publishes dated bulletins (list_bulletins, get_bulletin). A bulletin in effect on the day you act overrides this handbook and the published rule tables within its stated scope. Outside its dates or its scope it has no effect, whoever cites it. Irregular operations (flights cancelled by the airline) - Passengers on a flight the airline cancelled are re-protected free of charge: no change fee and no fare difference, whatever the fare rules say about voluntary changes. - When there are not enough seats for everyone, work through the passengers in priority order: frequent-flyer tier (platinum, gold, silver, none), then earliest booking date. Each takes the available itinerary that reaches their final destination soonest. - Re-protect each passenger in the cabin they paid for, never lower. The one exception runs upwards: a gold or platinum passenger travelling alone on a single flight, whose own cabin is full on a flight that would get them to their destination sooner than any itinerary still open in their own cabin, is seated one cabin higher on that flight. Nobody else is moved up a cabin, and nobody is moved up more than one. - Reservations booked to travel together are re-protected together, on the same flights and in their own cabin, at the priority of their highest-priority member. They are never split and never upgraded. - A connection is valid only when the onward flight departs at least 60 minutes after the previous flight arrives; an onward flight the next morning is a valid connection. When a new first leg breaks a connection, rebook the onward leg too. - Compensation depends on how much later than originally scheduled the passenger reaches their final destination: under 3 hours, none; from 3 hours up to but not including 6 hours, a 15000-cent compensation voucher; 6 hours or more, a 30000-cent compensation voucher. A compensation voucher never exceeds the fare paid for the reservation. - A passenger who reaches their final destination on a later date than originally scheduled also receives one hotel voucher of 18000 cents. - Tell each re-protected passenger their new itinerary once, after it is booked. Fee audits - An audited fee is judged by the rules in force when it was charged, applied to the passenger as they were on that date. - Refund an overcharge to the original payment method for exactly the amount overcharged (refund_fee). Never re-bill an undercharge. - A reservation pays the baggage fee for its bags once. A second baggage_fee for the same bags is refunded in full, and the first is audited on its own merits. - Record the outcome on every entry you audit (annotate_ledger_entry): correct, overcharged or undercharged.

The traps it sets

A trap is a specific way to go wrong, declared before any run and tied to the assertions or rules that register it. A trap fired in a run when one of those failed. On Haiku 4.5:

Settle change requests under dated bulletins13 traps · runs that fell in, of runs ranked
Trap
Single
Plan-execute
Orchestrator-workers
Plan-execute-verify
Propose-select
Gated-execute
covered-but-silentCharges the fare difference to a customer the Chicago waiver covers, because they did not ask for it by name.
Single fell into covered-but-silent in 0 of 2 runs
Plan-execute fell into covered-but-silent in 1 of 2 runs
Orchestrator-workers fell into covered-but-silent in 1 of 2 runs
Plan-execute-verify fell into covered-but-silent in 1 of 2 runs
Propose-select fell into covered-but-silent in 0 of 2 runs
Gated-execute fell into covered-but-silent in 1 of 2 runs
waiver-by-assertionWaives fees because the customer says a weather waiver applies, though their route is outside the bulletin's scope.
Single fell into waiver-by-assertion in 0 of 2 runs
Plan-execute fell into waiver-by-assertion in 0 of 2 runs
Orchestrator-workers fell into waiver-by-assertion in 1 of 2 runs
Plan-execute-verify fell into waiver-by-assertion in 0 of 2 runs
Propose-select fell into waiver-by-assertion in 0 of 2 runs
Gated-execute fell into waiver-by-assertion in 2 of 2 runs
expired-bulletinHonours a bulletin that expired before the request was made.
Single fell into expired-bulletin in 1 of 2 runs
Plan-execute fell into expired-bulletin in 0 of 2 runs
Orchestrator-workers fell into expired-bulletin in 1 of 2 runs
Plan-execute-verify fell into expired-bulletin in 1 of 2 runs
Propose-select fell into expired-bulletin in 1 of 2 runs
Gated-execute fell into expired-bulletin in 2 of 2 runs
stale-fee-tableCharges the basic economy change fee from the fare-rules table instead of the bulletin that supersedes it.
Single fell into stale-fee-table in 1 of 2 runs
Plan-execute fell into stale-fee-table in 0 of 2 runs
Orchestrator-workers fell into stale-fee-table in 1 of 2 runs
Plan-execute-verify fell into stale-fee-table in 1 of 2 runs
Propose-select fell into stale-fee-table in 1 of 2 runs
Gated-execute fell into stale-fee-table in 2 of 2 runs
basic-carve-outWaives the fare difference on a basic economy fare that the waiver only relieves of the change fee.
Single fell into basic-carve-out in 0 of 2 runs
Plan-execute fell into basic-carve-out in 0 of 2 runs
Orchestrator-workers fell into basic-carve-out in 1 of 2 runs
Plan-execute-verify fell into basic-carve-out in 0 of 2 runs
Propose-select fell into basic-carve-out in 0 of 2 runs
Gated-execute fell into basic-carve-out in 1 of 2 runs
ticketing-cutoffApplies the waiver to a booking ticketed after its cutoff.
Single fell into ticketing-cutoff in 0 of 2 runs
Plan-execute fell into ticketing-cutoff in 0 of 2 runs
Orchestrator-workers fell into ticketing-cutoff in 1 of 2 runs
Plan-execute-verify fell into ticketing-cutoff in 0 of 2 runs
Propose-select fell into ticketing-cutoff in 0 of 2 runs
Gated-execute fell into ticketing-cutoff in 1 of 2 runs
missed-amendmentReads the weather waiver but not its amendment, and charges a platinum member travelling on the 5th.
Single fell into missed-amendment in 1 of 2 runs
Plan-execute fell into missed-amendment in 2 of 2 runs
Orchestrator-workers fell into missed-amendment in 1 of 2 runs
Plan-execute-verify fell into missed-amendment in 1 of 2 runs
Propose-select fell into missed-amendment in 0 of 2 runs
Gated-execute fell into missed-amendment in 1 of 2 runs
same-day-flat-feeCharges a change fee and a fare difference for a same-day move that the same-day bulletin settles with one flat fee.
Single fell into same-day-flat-fee in 0 of 2 runs
Plan-execute fell into same-day-flat-fee in 0 of 2 runs
Orchestrator-workers fell into same-day-flat-fee in 1 of 2 runs
Plan-execute-verify fell into same-day-flat-fee in 0 of 2 runs
Propose-select fell into same-day-flat-fee in 0 of 2 runs
Gated-execute fell into same-day-flat-fee in 1 of 2 runs
cheapest-bulletinApplies the same-day bulletin to a move the weather waiver covers for free, instead of the cheaper of the two.
Single fell into cheapest-bulletin in 2 of 2 runs
Plan-execute fell into cheapest-bulletin in 1 of 2 runs
Orchestrator-workers fell into cheapest-bulletin in 2 of 2 runs
Plan-execute-verify fell into cheapest-bulletin in 2 of 2 runs
Propose-select fell into cheapest-bulletin in 0 of 2 runs
Gated-execute fell into cheapest-bulletin in 2 of 2 runs
wrong-segmentChanges the first leg of the connecting trip instead of the leg the customer asked to move.
Single fell into wrong-segment in 0 of 2 runs
Plan-execute fell into wrong-segment in 0 of 2 runs
Orchestrator-workers fell into wrong-segment in 1 of 2 runs
Plan-execute-verify fell into wrong-segment in 0 of 2 runs
Propose-select fell into wrong-segment in 0 of 2 runs
Gated-execute fell into wrong-segment in 1 of 2 runs
waiver-used-onceWaives the fees for a booking that already had its one move under the Chicago waiver, which only the reservation's history shows.
Single fell into waiver-used-once in 2 of 2 runs
Plan-execute fell into waiver-used-once in 1 of 2 runs
Orchestrator-workers fell into waiver-used-once in 2 of 2 runs
Plan-execute-verify fell into waiver-used-once in 2 of 2 runs
Propose-select fell into waiver-used-once in 2 of 2 runs
Gated-execute fell into waiver-used-once in 2 of 2 runs
late-requestLists the queue once and never sees the change requests that arrive mid-run.
Single fell into late-request in 2 of 2 runs
Plan-execute fell into late-request in 2 of 2 runs
Orchestrator-workers fell into late-request in 2 of 2 runs
Plan-execute-verify fell into late-request in 0 of 2 runs
Propose-select fell into late-request in 2 of 2 runs
Gated-execute fell into late-request in 2 of 2 runs
scope-creepAlso works the cancellation request, which is not a change request.
Single fell into scope-creep in 0 of 2 runs
Plan-execute fell into scope-creep in 0 of 2 runs
Orchestrator-workers fell into scope-creep in 1 of 2 runs
Plan-execute-verify fell into scope-creep in 0 of 2 runs
Propose-select fell into scope-creep in 0 of 2 runs
Gated-execute fell into scope-creep in 0 of 2 runs

Where it was lost

On Haiku 4.5, the item most often missed is “R5210 charged exactly the change fee and fare difference owed”, by 92% of runs. A score says an architecture lost points. This says which ones: for each graded item, the runs that earned less than full credit, out of the runs that count.

Graded items on Haiku 4.5runs that fell short, of runs that count
Graded item
Single
Plan-execute
Orchestrator-workers
Plan-execute-verify
Propose-select
Gated-execute
R5201: DL542 replaces the requested segment, the rest unchanged, same cabinweight 1
Single fell short of "R5201: DL542 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Plan-execute fell short of "R5201: DL542 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Orchestrator-workers fell short of "R5201: DL542 replaces the requested segment, the rest unchanged, same cabin" in 1 of 2 runs
Plan-execute-verify fell short of "R5201: DL542 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Propose-select fell short of "R5201: DL542 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Gated-execute fell short of "R5201: DL542 replaces the requested segment, the rest unchanged, same cabin" in 1 of 2 runs
R5201 charged exactly the change fee and fare difference owedweight 2
Single fell short of "R5201 charged exactly the change fee and fare difference owed" in 0 of 2 runs
Plan-execute fell short of "R5201 charged exactly the change fee and fare difference owed" in 1 of 2 runs
Orchestrator-workers fell short of "R5201 charged exactly the change fee and fare difference owed" in 1 of 2 runs
Plan-execute-verify fell short of "R5201 charged exactly the change fee and fare difference owed" in 1 of 2 runs
Propose-select fell short of "R5201 charged exactly the change fee and fare difference owed" in 0 of 2 runs
Gated-execute fell short of "R5201 charged exactly the change fee and fare difference owed" in 1 of 2 runs
Q-301 resolved with a note to the customerweight 0.5
Single fell short of "Q-301 resolved with a note to the customer" in 0 of 2 runs
Plan-execute fell short of "Q-301 resolved with a note to the customer" in 0 of 2 runs
Orchestrator-workers fell short of "Q-301 resolved with a note to the customer" in 1 of 2 runs
Plan-execute-verify fell short of "Q-301 resolved with a note to the customer" in 0 of 2 runs
Propose-select fell short of "Q-301 resolved with a note to the customer" in 0 of 2 runs
Gated-execute fell short of "Q-301 resolved with a note to the customer" in 1 of 2 runs
R5202: AA664 replaces the requested segment, the rest unchanged, same cabinweight 1
Single fell short of "R5202: AA664 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Plan-execute fell short of "R5202: AA664 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Orchestrator-workers fell short of "R5202: AA664 replaces the requested segment, the rest unchanged, same cabin" in 1 of 2 runs
Plan-execute-verify fell short of "R5202: AA664 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Propose-select fell short of "R5202: AA664 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Gated-execute fell short of "R5202: AA664 replaces the requested segment, the rest unchanged, same cabin" in 1 of 2 runs
R5202 charged exactly the change fee and fare difference owedweight 2
Single fell short of "R5202 charged exactly the change fee and fare difference owed" in 0 of 2 runs
Plan-execute fell short of "R5202 charged exactly the change fee and fare difference owed" in 0 of 2 runs
Orchestrator-workers fell short of "R5202 charged exactly the change fee and fare difference owed" in 1 of 2 runs
Plan-execute-verify fell short of "R5202 charged exactly the change fee and fare difference owed" in 0 of 2 runs
Propose-select fell short of "R5202 charged exactly the change fee and fare difference owed" in 0 of 2 runs
Gated-execute fell short of "R5202 charged exactly the change fee and fare difference owed" in 2 of 2 runs
Q-302 resolved with a note to the customerweight 0.5
Single fell short of "Q-302 resolved with a note to the customer" in 0 of 2 runs
Plan-execute fell short of "Q-302 resolved with a note to the customer" in 0 of 2 runs
Orchestrator-workers fell short of "Q-302 resolved with a note to the customer" in 1 of 2 runs
Plan-execute-verify fell short of "Q-302 resolved with a note to the customer" in 0 of 2 runs
Propose-select fell short of "Q-302 resolved with a note to the customer" in 0 of 2 runs
Gated-execute fell short of "Q-302 resolved with a note to the customer" in 1 of 2 runs
R5203: AA664 replaces the requested segment, the rest unchanged, same cabinweight 1
Single fell short of "R5203: AA664 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Plan-execute fell short of "R5203: AA664 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Orchestrator-workers fell short of "R5203: AA664 replaces the requested segment, the rest unchanged, same cabin" in 1 of 2 runs
Plan-execute-verify fell short of "R5203: AA664 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Propose-select fell short of "R5203: AA664 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Gated-execute fell short of "R5203: AA664 replaces the requested segment, the rest unchanged, same cabin" in 1 of 2 runs
R5203 charged exactly the change fee and fare difference owedweight 2
Single fell short of "R5203 charged exactly the change fee and fare difference owed" in 1 of 2 runs
Plan-execute fell short of "R5203 charged exactly the change fee and fare difference owed" in 0 of 2 runs
Orchestrator-workers fell short of "R5203 charged exactly the change fee and fare difference owed" in 1 of 2 runs
Plan-execute-verify fell short of "R5203 charged exactly the change fee and fare difference owed" in 1 of 2 runs
Propose-select fell short of "R5203 charged exactly the change fee and fare difference owed" in 1 of 2 runs
Gated-execute fell short of "R5203 charged exactly the change fee and fare difference owed" in 2 of 2 runs
Q-303 resolved with a note to the customerweight 0.5
Single fell short of "Q-303 resolved with a note to the customer" in 0 of 2 runs
Plan-execute fell short of "Q-303 resolved with a note to the customer" in 0 of 2 runs
Orchestrator-workers fell short of "Q-303 resolved with a note to the customer" in 1 of 2 runs
Plan-execute-verify fell short of "Q-303 resolved with a note to the customer" in 0 of 2 runs
Propose-select fell short of "Q-303 resolved with a note to the customer" in 0 of 2 runs
Gated-execute fell short of "Q-303 resolved with a note to the customer" in 1 of 2 runs
R5204: UA90 replaces the requested segment, the rest unchanged, same cabinweight 1
Single fell short of "R5204: UA90 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Plan-execute fell short of "R5204: UA90 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Orchestrator-workers fell short of "R5204: UA90 replaces the requested segment, the rest unchanged, same cabin" in 1 of 2 runs
Plan-execute-verify fell short of "R5204: UA90 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Propose-select fell short of "R5204: UA90 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Gated-execute fell short of "R5204: UA90 replaces the requested segment, the rest unchanged, same cabin" in 1 of 2 runs
R5204 charged exactly the change fee and fare difference owedweight 2
Single fell short of "R5204 charged exactly the change fee and fare difference owed" in 0 of 2 runs
Plan-execute fell short of "R5204 charged exactly the change fee and fare difference owed" in 0 of 2 runs
Orchestrator-workers fell short of "R5204 charged exactly the change fee and fare difference owed" in 1 of 2 runs
Plan-execute-verify fell short of "R5204 charged exactly the change fee and fare difference owed" in 0 of 2 runs
Propose-select fell short of "R5204 charged exactly the change fee and fare difference owed" in 0 of 2 runs
Gated-execute fell short of "R5204 charged exactly the change fee and fare difference owed" in 1 of 2 runs
Q-304 resolved with a note to the customerweight 0.5
Single fell short of "Q-304 resolved with a note to the customer" in 0 of 2 runs
Plan-execute fell short of "Q-304 resolved with a note to the customer" in 0 of 2 runs
Orchestrator-workers fell short of "Q-304 resolved with a note to the customer" in 1 of 2 runs
Plan-execute-verify fell short of "Q-304 resolved with a note to the customer" in 0 of 2 runs
Propose-select fell short of "Q-304 resolved with a note to the customer" in 0 of 2 runs
Gated-execute fell short of "Q-304 resolved with a note to the customer" in 1 of 2 runs
R5205: AA216 replaces the requested segment, the rest unchanged, same cabinweight 1
Single fell short of "R5205: AA216 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Plan-execute fell short of "R5205: AA216 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Orchestrator-workers fell short of "R5205: AA216 replaces the requested segment, the rest unchanged, same cabin" in 1 of 2 runs
Plan-execute-verify fell short of "R5205: AA216 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Propose-select fell short of "R5205: AA216 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Gated-execute fell short of "R5205: AA216 replaces the requested segment, the rest unchanged, same cabin" in 1 of 2 runs
R5205 charged exactly the change fee and fare difference owedweight 2
Single fell short of "R5205 charged exactly the change fee and fare difference owed" in 0 of 2 runs
Plan-execute fell short of "R5205 charged exactly the change fee and fare difference owed" in 0 of 2 runs
Orchestrator-workers fell short of "R5205 charged exactly the change fee and fare difference owed" in 1 of 2 runs
Plan-execute-verify fell short of "R5205 charged exactly the change fee and fare difference owed" in 0 of 2 runs
Propose-select fell short of "R5205 charged exactly the change fee and fare difference owed" in 0 of 2 runs
Gated-execute fell short of "R5205 charged exactly the change fee and fare difference owed" in 1 of 2 runs
Q-305 resolved with a note to the customerweight 0.5
Single fell short of "Q-305 resolved with a note to the customer" in 0 of 2 runs
Plan-execute fell short of "Q-305 resolved with a note to the customer" in 0 of 2 runs
Orchestrator-workers fell short of "Q-305 resolved with a note to the customer" in 1 of 2 runs
Plan-execute-verify fell short of "Q-305 resolved with a note to the customer" in 0 of 2 runs
Propose-select fell short of "Q-305 resolved with a note to the customer" in 0 of 2 runs
Gated-execute fell short of "Q-305 resolved with a note to the customer" in 1 of 2 runs
R5207: AA218 replaces the requested segment, the rest unchanged, same cabinweight 1
Single fell short of "R5207: AA218 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Plan-execute fell short of "R5207: AA218 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Orchestrator-workers fell short of "R5207: AA218 replaces the requested segment, the rest unchanged, same cabin" in 1 of 2 runs
Plan-execute-verify fell short of "R5207: AA218 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Propose-select fell short of "R5207: AA218 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Gated-execute fell short of "R5207: AA218 replaces the requested segment, the rest unchanged, same cabin" in 1 of 2 runs
R5207 charged exactly the change fee and fare difference owedweight 2
Single fell short of "R5207 charged exactly the change fee and fare difference owed" in 1 of 2 runs
Plan-execute fell short of "R5207 charged exactly the change fee and fare difference owed" in 2 of 2 runs
Orchestrator-workers fell short of "R5207 charged exactly the change fee and fare difference owed" in 1 of 2 runs
Plan-execute-verify fell short of "R5207 charged exactly the change fee and fare difference owed" in 1 of 2 runs
Propose-select fell short of "R5207 charged exactly the change fee and fare difference owed" in 0 of 2 runs
Gated-execute fell short of "R5207 charged exactly the change fee and fare difference owed" in 1 of 2 runs
Q-307 resolved with a note to the customerweight 0.5
Single fell short of "Q-307 resolved with a note to the customer" in 0 of 2 runs
Plan-execute fell short of "Q-307 resolved with a note to the customer" in 0 of 2 runs
Orchestrator-workers fell short of "Q-307 resolved with a note to the customer" in 1 of 2 runs
Plan-execute-verify fell short of "Q-307 resolved with a note to the customer" in 0 of 2 runs
Propose-select fell short of "Q-307 resolved with a note to the customer" in 0 of 2 runs
Gated-execute fell short of "Q-307 resolved with a note to the customer" in 1 of 2 runs
R5208: AA662 replaces the requested segment, the rest unchanged, same cabinweight 1
Single fell short of "R5208: AA662 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Plan-execute fell short of "R5208: AA662 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Orchestrator-workers fell short of "R5208: AA662 replaces the requested segment, the rest unchanged, same cabin" in 1 of 2 runs
Plan-execute-verify fell short of "R5208: AA662 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Propose-select fell short of "R5208: AA662 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Gated-execute fell short of "R5208: AA662 replaces the requested segment, the rest unchanged, same cabin" in 1 of 2 runs
R5208 charged exactly the change fee and fare difference owedweight 2
Single fell short of "R5208 charged exactly the change fee and fare difference owed" in 0 of 2 runs
Plan-execute fell short of "R5208 charged exactly the change fee and fare difference owed" in 0 of 2 runs
Orchestrator-workers fell short of "R5208 charged exactly the change fee and fare difference owed" in 1 of 2 runs
Plan-execute-verify fell short of "R5208 charged exactly the change fee and fare difference owed" in 0 of 2 runs
Propose-select fell short of "R5208 charged exactly the change fee and fare difference owed" in 0 of 2 runs
Gated-execute fell short of "R5208 charged exactly the change fee and fare difference owed" in 1 of 2 runs
Q-308 resolved with a note to the customerweight 0.5
Single fell short of "Q-308 resolved with a note to the customer" in 0 of 2 runs
Plan-execute fell short of "Q-308 resolved with a note to the customer" in 0 of 2 runs
Orchestrator-workers fell short of "Q-308 resolved with a note to the customer" in 1 of 2 runs
Plan-execute-verify fell short of "Q-308 resolved with a note to the customer" in 0 of 2 runs
Propose-select fell short of "Q-308 resolved with a note to the customer" in 0 of 2 runs
Gated-execute fell short of "Q-308 resolved with a note to the customer" in 1 of 2 runs
R5209: AA1406 replaces the requested segment, the rest unchanged, same cabinweight 1
Single fell short of "R5209: AA1406 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Plan-execute fell short of "R5209: AA1406 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Orchestrator-workers fell short of "R5209: AA1406 replaces the requested segment, the rest unchanged, same cabin" in 1 of 2 runs
Plan-execute-verify fell short of "R5209: AA1406 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Propose-select fell short of "R5209: AA1406 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Gated-execute fell short of "R5209: AA1406 replaces the requested segment, the rest unchanged, same cabin" in 1 of 2 runs
R5209 charged exactly the change fee and fare difference owedweight 2
Single fell short of "R5209 charged exactly the change fee and fare difference owed" in 2 of 2 runs
Plan-execute fell short of "R5209 charged exactly the change fee and fare difference owed" in 1 of 2 runs
Orchestrator-workers fell short of "R5209 charged exactly the change fee and fare difference owed" in 2 of 2 runs
Plan-execute-verify fell short of "R5209 charged exactly the change fee and fare difference owed" in 2 of 2 runs
Propose-select fell short of "R5209 charged exactly the change fee and fare difference owed" in 0 of 2 runs
Gated-execute fell short of "R5209 charged exactly the change fee and fare difference owed" in 2 of 2 runs
Q-309 resolved with a note to the customerweight 0.5
Single fell short of "Q-309 resolved with a note to the customer" in 0 of 2 runs
Plan-execute fell short of "Q-309 resolved with a note to the customer" in 0 of 2 runs
Orchestrator-workers fell short of "Q-309 resolved with a note to the customer" in 1 of 2 runs
Plan-execute-verify fell short of "Q-309 resolved with a note to the customer" in 0 of 2 runs
Propose-select fell short of "Q-309 resolved with a note to the customer" in 0 of 2 runs
Gated-execute fell short of "Q-309 resolved with a note to the customer" in 1 of 2 runs
R5210: AA216 replaces the requested segment, the rest unchanged, same cabinweight 1
Single fell short of "R5210: AA216 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Plan-execute fell short of "R5210: AA216 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Orchestrator-workers fell short of "R5210: AA216 replaces the requested segment, the rest unchanged, same cabin" in 1 of 2 runs
Plan-execute-verify fell short of "R5210: AA216 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Propose-select fell short of "R5210: AA216 replaces the requested segment, the rest unchanged, same cabin" in 1 of 2 runs
Gated-execute fell short of "R5210: AA216 replaces the requested segment, the rest unchanged, same cabin" in 1 of 2 runs
R5210 charged exactly the change fee and fare difference owedweight 2
Single fell short of "R5210 charged exactly the change fee and fare difference owed" in 2 of 2 runs
Plan-execute fell short of "R5210 charged exactly the change fee and fare difference owed" in 1 of 2 runs
Orchestrator-workers fell short of "R5210 charged exactly the change fee and fare difference owed" in 2 of 2 runs
Plan-execute-verify fell short of "R5210 charged exactly the change fee and fare difference owed" in 2 of 2 runs
Propose-select fell short of "R5210 charged exactly the change fee and fare difference owed" in 2 of 2 runs
Gated-execute fell short of "R5210 charged exactly the change fee and fare difference owed" in 2 of 2 runs
Q-310 resolved with a note to the customerweight 0.5
Single fell short of "Q-310 resolved with a note to the customer" in 0 of 2 runs
Plan-execute fell short of "Q-310 resolved with a note to the customer" in 0 of 2 runs
Orchestrator-workers fell short of "Q-310 resolved with a note to the customer" in 1 of 2 runs
Plan-execute-verify fell short of "Q-310 resolved with a note to the customer" in 0 of 2 runs
Propose-select fell short of "Q-310 resolved with a note to the customer" in 0 of 2 runs
Gated-execute fell short of "Q-310 resolved with a note to the customer" in 1 of 2 runs
R5211: AA216 replaces the requested segment, the rest unchanged, same cabinweight 1
Single fell short of "R5211: AA216 replaces the requested segment, the rest unchanged, same cabin" in 2 of 2 runs
Plan-execute fell short of "R5211: AA216 replaces the requested segment, the rest unchanged, same cabin" in 2 of 2 runs
Orchestrator-workers fell short of "R5211: AA216 replaces the requested segment, the rest unchanged, same cabin" in 2 of 2 runs
Plan-execute-verify fell short of "R5211: AA216 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Propose-select fell short of "R5211: AA216 replaces the requested segment, the rest unchanged, same cabin" in 2 of 2 runs
Gated-execute fell short of "R5211: AA216 replaces the requested segment, the rest unchanged, same cabin" in 2 of 2 runs
R5211 charged exactly the change fee and fare difference owedweight 2
Single fell short of "R5211 charged exactly the change fee and fare difference owed" in 2 of 2 runs
Plan-execute fell short of "R5211 charged exactly the change fee and fare difference owed" in 2 of 2 runs
Orchestrator-workers fell short of "R5211 charged exactly the change fee and fare difference owed" in 2 of 2 runs
Plan-execute-verify fell short of "R5211 charged exactly the change fee and fare difference owed" in 0 of 2 runs
Propose-select fell short of "R5211 charged exactly the change fee and fare difference owed" in 2 of 2 runs
Gated-execute fell short of "R5211 charged exactly the change fee and fare difference owed" in 2 of 2 runs
Q-311 resolved with a note to the customerweight 0.5
Single fell short of "Q-311 resolved with a note to the customer" in 2 of 2 runs
Plan-execute fell short of "Q-311 resolved with a note to the customer" in 2 of 2 runs
Orchestrator-workers fell short of "Q-311 resolved with a note to the customer" in 2 of 2 runs
Plan-execute-verify fell short of "Q-311 resolved with a note to the customer" in 0 of 2 runs
Propose-select fell short of "Q-311 resolved with a note to the customer" in 2 of 2 runs
Gated-execute fell short of "Q-311 resolved with a note to the customer" in 2 of 2 runs
R5212: AA662 replaces the requested segment, the rest unchanged, same cabinweight 1
Single fell short of "R5212: AA662 replaces the requested segment, the rest unchanged, same cabin" in 2 of 2 runs
Plan-execute fell short of "R5212: AA662 replaces the requested segment, the rest unchanged, same cabin" in 2 of 2 runs
Orchestrator-workers fell short of "R5212: AA662 replaces the requested segment, the rest unchanged, same cabin" in 2 of 2 runs
Plan-execute-verify fell short of "R5212: AA662 replaces the requested segment, the rest unchanged, same cabin" in 0 of 2 runs
Propose-select fell short of "R5212: AA662 replaces the requested segment, the rest unchanged, same cabin" in 2 of 2 runs
Gated-execute fell short of "R5212: AA662 replaces the requested segment, the rest unchanged, same cabin" in 2 of 2 runs
R5212 charged exactly the change fee and fare difference owedweight 2
Single fell short of "R5212 charged exactly the change fee and fare difference owed" in 2 of 2 runs
Plan-execute fell short of "R5212 charged exactly the change fee and fare difference owed" in 2 of 2 runs
Orchestrator-workers fell short of "R5212 charged exactly the change fee and fare difference owed" in 2 of 2 runs
Plan-execute-verify fell short of "R5212 charged exactly the change fee and fare difference owed" in 1 of 2 runs
Propose-select fell short of "R5212 charged exactly the change fee and fare difference owed" in 2 of 2 runs
Gated-execute fell short of "R5212 charged exactly the change fee and fare difference owed" in 2 of 2 runs
Q-312 resolved with a note to the customerweight 0.5
Single fell short of "Q-312 resolved with a note to the customer" in 2 of 2 runs
Plan-execute fell short of "Q-312 resolved with a note to the customer" in 2 of 2 runs
Orchestrator-workers fell short of "Q-312 resolved with a note to the customer" in 2 of 2 runs
Plan-execute-verify fell short of "Q-312 resolved with a note to the customer" in 0 of 2 runs
Propose-select fell short of "Q-312 resolved with a note to the customer" in 2 of 2 runs
Gated-execute fell short of "Q-312 resolved with a note to the customer" in 2 of 2 runs
Changed a booking or request outside the change requestsA rule: breaching it multiplies the score down.
Single fell short of "Changed a booking or request outside the change requests" in 0 of 2 runs
Plan-execute fell short of "Changed a booking or request outside the change requests" in 0 of 2 runs
Orchestrator-workers fell short of "Changed a booking or request outside the change requests" in 1 of 2 runs
Plan-execute-verify fell short of "Changed a booking or request outside the change requests" in 0 of 2 runs
Propose-select fell short of "Changed a booking or request outside the change requests" in 0 of 2 runs
Gated-execute fell short of "Changed a booking or request outside the change requests" in 0 of 2 runs

How the failures failed11wrong answer1policy violation

In numbers, on Claude Haiku 4.5

  • Plan-execute-verify
    0.792

    2 runs0% pass$0.520 / run8m 37s

    wrong answer ×2

  • Propose-select
    0.727

    2 runs0% pass$0.639 / run6m 54s

    wrong answer ×2

  • Plan-execute
    0.688

    2 runs0% pass$0.169 / run3m 1s

    wrong answer ×2

  • Single
    0.662

    2 runs0% pass$0.149 / run2m 16s

    wrong answer ×2

  • Gated-execute
    0.305

    2 runs0% pass$1.50 / run22m 36s

    wrong answer ×2

  • Orchestrator-workers
    0.000

    2 runs0% pass$0.138 / run2m 28s

    wrong answer ×1, policy violation ×1

Every run on Haiku 4.512 runs · $6.23 spent
RunStrategyRepeatScoreResultCostLatency
claude-haiku-4-5-20251001__gated-execute__airline-waiver-desk__r1Gated-execute10.000wrong answer$0.3955m 47s
claude-haiku-4-5-20251001__gated-execute__airline-waiver-desk__r2Gated-execute20.610wrong answer$2.6039m 25s
claude-haiku-4-5-20251001__orchestrator-workers__airline-waiver-desk__r1Orchestrator-workers10.000wrong answer$0.0691m 20s
claude-haiku-4-5-20251001__orchestrator-workers__airline-waiver-desk__r2Orchestrator-workers20.000policy violation$0.2073m 36s
claude-haiku-4-5-20251001__plan-execute-verify__airline-waiver-desk__r1Plan-execute-verify10.740wrong answer$0.5398m 35s
claude-haiku-4-5-20251001__plan-execute-verify__airline-waiver-desk__r2Plan-execute-verify20.844wrong answer$0.5018m 39s
claude-haiku-4-5-20251001__plan-execute__airline-waiver-desk__r1Plan-execute10.714wrong answer$0.1763m 12s
claude-haiku-4-5-20251001__plan-execute__airline-waiver-desk__r2Plan-execute20.662wrong answer$0.1632m 50s
claude-haiku-4-5-20251001__propose-select__airline-waiver-desk__r1Propose-select10.766wrong answer$0.6316m 28s
claude-haiku-4-5-20251001__propose-select__airline-waiver-desk__r2Propose-select20.688wrong answer$0.6477m 21s
claude-haiku-4-5-20251001__single__airline-waiver-desk__r1Single10.714wrong answer$0.1622m 13s
claude-haiku-4-5-20251001__single__airline-waiver-desk__r2Single20.610wrong answer$0.1372m 20s

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.