ScaffoldBench

tool-use · airline

Audit a week of baggage fees

expertauditbatchindependent-itemspartial-observabilitypagination
Task id
airline-baggage-audit
Turn limit
140
Runs on Sonnet 5.5
12
Pass rate
17%

Pre-registered hypothesis

Hypothesis

orchestrator-workers ≈ single on score at lower latency; plan-execute-verify highest. Nineteen independent charges across three ledger pages: fan-out parallelises the per-entry lookups, while a single context drifts or stops early. Each finding is checkable from state (tier and booking history, the dated fee bulletin, earlier refunds on the ledger), so a state-grounded verifier catches omissions and refunds computed from today's tier, today's cabin or yesterday's price list.

Written when the task was written, before any run. It is a prediction in words, so read it against what happened rather than trusting a verdict drawn for you.

What happened on Sonnet 5.5

  • Highest mean score: Propose-select 1.000
  • A single agent: 0.702
  • Lowest: Plan-execute-verify 0.358

Means over the runs that count; intervals and verdicts are on the results page.

Who solved it

On Sonnet 5.5, Propose-select scores highest at 1.000 and Plan-execute-verify lowest at 0.358. The same task, every architecture, every model: one square per run.

0.435 mean score of the runs that count
0.702 mean score of the runs that count
0.772 mean score of the runs that count
0.540 mean score of the runs that count
0.716 mean score of the runs that count
0.844 mean score of the runs that count
0.428 mean score of the runs that count
0.688 mean score of the runs that count
0.500 mean score of the runs that count
0.316 mean score of the runs that count
0.358 mean score of the runs that count
0.886 mean score of the runs that count
0.260 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
0.000 mean score of the runs that count
0.751 mean score of the runs that count
1.000 mean score of the runs that count

Point at a square or tab to it to read its run. Each square opens its run.

passedpartial creditfailedexcluded: integrity check failedexcluded: API errorOne square per run; each opens its run.

The prompt every architecture was handed

Finance thinks some baggage fees posted from 2026-07-20 through 2026-07-26 used the wrong allowances or prices. Audit every baggage fee posted in that window and settle each one per the audit rules.

The domain policy

Appended to the system prompt for every architecture, so none of them gets a rules advantage.

Reservations desk handbook. You act for the airline. These rules bind you even when a customer asks for something else. General - Act only on the reservations a request is about. Do not change any other booking, passenger record or flight. - Money moves only through the ledger tools. Each fee, refund or close-out is its own ledger entry. - Tell the customer plainly what they are getting and why, especially when it is not what they asked for. Checked baggage - The free allowance and the fee schedule are published in the baggage rules (get_baggage_rules). Read them before charging any baggage fee. - Charge only for bags beyond the free allowance, at the published prices, as a single baggage_fee entry. - A reservation may hold at most 3 checked bags. Voluntary flight changes - Change fees, who is exempt from them, and how fare differences are settled are set out in the fare rules for the cabin (get_fare_rules). Read them before changing a flight. - Confirm the status and seat availability of a flight (search_flights or get_flight) before booking anyone onto it. The booking tool is not a search tool. - Keep the cabin the customer booked unless they ask for a different one. Cancellations and refunds - Read the refund policy for the reservation (get_refund_policy) before moving any money for it. - Money goes back to the original form of payment only when the refund policy makes the booking eligible for that. Otherwise settle it the way the policy says: travel credit, or a non-refundable record. - Every cancelled reservation is closed out financially, with a refund or with a non-refundable record. Request queue - Customer requests arrive in the service queue (list_requests, get_request). Work the requests you are asked to work and leave the rest for the teams that own them. - Resolve each request you complete (resolve_request) with a short note to the customer saying what was done and what they are getting. Payments - Money can only be sent to a payment instrument that is active. When the instrument a booking was paid with is no longer active, issue the amount as travel credit instead and tell the customer why. Operational bulletins - Operations publishes dated bulletins (list_bulletins, get_bulletin). A bulletin in effect on the day you act overrides this handbook and the published rule tables within its stated scope. Outside its dates or its scope it has no effect, whoever cites it. Irregular operations (flights cancelled by the airline) - Passengers on a flight the airline cancelled are re-protected free of charge: no change fee and no fare difference, whatever the fare rules say about voluntary changes. - When there are not enough seats for everyone, work through the passengers in priority order: frequent-flyer tier (platinum, gold, silver, none), then earliest booking date. Each takes the available itinerary that reaches their final destination soonest. - Re-protect each passenger in the cabin they paid for, never lower. The one exception runs upwards: a gold or platinum passenger travelling alone on a single flight, whose own cabin is full on a flight that would get them to their destination sooner than any itinerary still open in their own cabin, is seated one cabin higher on that flight. Nobody else is moved up a cabin, and nobody is moved up more than one. - Reservations booked to travel together are re-protected together, on the same flights and in their own cabin, at the priority of their highest-priority member. They are never split and never upgraded. - A connection is valid only when the onward flight departs at least 60 minutes after the previous flight arrives; an onward flight the next morning is a valid connection. When a new first leg breaks a connection, rebook the onward leg too. - Compensation depends on how much later than originally scheduled the passenger reaches their final destination: under 3 hours, none; from 3 hours up to but not including 6 hours, a 15000-cent compensation voucher; 6 hours or more, a 30000-cent compensation voucher. A compensation voucher never exceeds the fare paid for the reservation. - A passenger who reaches their final destination on a later date than originally scheduled also receives one hotel voucher of 18000 cents. - Tell each re-protected passenger their new itinerary once, after it is booked. Fee audits - An audited fee is judged by the rules in force when it was charged, applied to the passenger as they were on that date. - Refund an overcharge to the original payment method for exactly the amount overcharged (refund_fee). Never re-bill an undercharge. - A reservation pays the baggage fee for its bags once. A second baggage_fee for the same bags is refunded in full, and the first is audited on its own merits. - Record the outcome on every entry you audit (annotate_ledger_entry): correct, overcharged or undercharged.

The traps it sets

A trap is a specific way to go wrong, declared before any run and tied to the assertions or rules that register it. A trap fired in a run when one of those failed. On Sonnet 5.5:

Audit a week of baggage fees9 traps · runs that fell in, of runs ranked
Trap
Single
Plan-execute
Orchestrator-workers
Plan-execute-verify
Propose-select
Gated-execute
current-tierJudges a charge by the passenger's tier today instead of on the day of the charge, refunding a correct fee or missing an overcharge.
Single fell into current-tier in 0 of 2 runs
Plan-execute fell into current-tier in 0 of 2 runs
Orchestrator-workers fell into current-tier in 0 of 2 runs
Plan-execute-verify fell into current-tier in 1 of 2 runs
Propose-select fell into current-tier in 0 of 2 runs
Gated-execute fell into current-tier in 0 of 2 runs
cabin-only-allowanceAccepts charges that ignored the tier bonus to the allowance.
Single fell into cabin-only-allowance in 0 of 2 runs
Plan-execute fell into cabin-only-allowance in 0 of 2 runs
Orchestrator-workers fell into cabin-only-allowance in 0 of 2 runs
Plan-execute-verify fell into cabin-only-allowance in 1 of 2 runs
Propose-select fell into cabin-only-allowance in 0 of 2 runs
Gated-execute fell into cabin-only-allowance in 0 of 2 runs
stale-fee-tablePrices bags from the baggage rules tool for charges posted after the fee bulletin took effect.
Single fell into stale-fee-table in 0 of 2 runs
Plan-execute fell into stale-fee-table in 0 of 2 runs
Orchestrator-workers fell into stale-fee-table in 0 of 2 runs
Plan-execute-verify fell into stale-fee-table in 0 of 2 runs
Propose-select fell into stale-fee-table in 0 of 2 runs
Gated-execute fell into stale-fee-table in 0 of 2 runs
later-upgradeJudges a charge by the booking's cabin today, after an upgrade, instead of the cabin it had when charged.
Single fell into later-upgrade in 2 of 2 runs
Plan-execute fell into later-upgrade in 2 of 2 runs
Orchestrator-workers fell into later-upgrade in 2 of 2 runs
Plan-execute-verify fell into later-upgrade in 1 of 2 runs
Propose-select fell into later-upgrade in 0 of 2 runs
Gated-execute fell into later-upgrade in 2 of 2 runs
duplicate-chargeMisses that one reservation was billed twice for the same bags.
Single fell into duplicate-charge in 0 of 2 runs
Plan-execute fell into duplicate-charge in 0 of 2 runs
Orchestrator-workers fell into duplicate-charge in 0 of 2 runs
Plan-execute-verify fell into duplicate-charge in 1 of 2 runs
Propose-select fell into duplicate-charge in 0 of 2 runs
Gated-execute fell into duplicate-charge in 0 of 2 runs
second-purchasePrices a bag added later in the week as a first chargeable bag, treating a correct charge as an overcharge.
Single fell into second-purchase in 0 of 2 runs
Plan-execute fell into second-purchase in 0 of 2 runs
Orchestrator-workers fell into second-purchase in 0 of 2 runs
Plan-execute-verify fell into second-purchase in 0 of 2 runs
Propose-select fell into second-purchase in 0 of 2 runs
Gated-execute fell into second-purchase in 0 of 2 runs
prior-courtesy-refundRefunds the whole overcharge without netting off a colleague's earlier courtesy refund.
Single fell into prior-courtesy-refund in 1 of 2 runs
Plan-execute fell into prior-courtesy-refund in 0 of 2 runs
Orchestrator-workers fell into prior-courtesy-refund in 2 of 2 runs
Plan-execute-verify fell into prior-courtesy-refund in 1 of 2 runs
Propose-select fell into prior-courtesy-refund in 0 of 2 runs
Gated-execute fell into prior-courtesy-refund in 1 of 2 runs
second-pageStops after the first page of ledger results.
Single fell into second-page in 0 of 2 runs
Plan-execute fell into second-page in 0 of 2 runs
Orchestrator-workers fell into second-page in 0 of 2 runs
Plan-execute-verify fell into second-page in 0 of 2 runs
Propose-select fell into second-page in 0 of 2 runs
Gated-execute fell into second-page in 0 of 2 runs
rebillCharges the customer again for an undercharge, or refunds an entry outside the window.
Single fell into rebill in 0 of 2 runs
Plan-execute fell into rebill in 0 of 2 runs
Orchestrator-workers fell into rebill in 0 of 2 runs
Plan-execute-verify fell into rebill in 1 of 2 runs
Propose-select fell into rebill in 0 of 2 runs
Gated-execute fell into rebill in 0 of 2 runs

Where it was lost

On Sonnet 5.5, the item most often missed is “L8 (R8120): audited, and refunded exactly the overcharge, if any”, by 75% of runs. A score says an architecture lost points. This says which ones: for each graded item, the runs that earned less than full credit, out of the runs that count.

Graded items on Sonnet 5.5runs that fell short, of runs that count
Graded item
Single
Plan-execute
Orchestrator-workers
Plan-execute-verify
Propose-select
Gated-execute
L2 (R8124): audited, and refunded exactly the overcharge, if anyweight 1
Single fell short of "L2 (R8124): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Plan-execute fell short of "L2 (R8124): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Orchestrator-workers fell short of "L2 (R8124): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Plan-execute-verify fell short of "L2 (R8124): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Propose-select fell short of "L2 (R8124): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Gated-execute fell short of "L2 (R8124): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
L2: audit outcome recorded as correctweight 0.5
Single fell short of "L2: audit outcome recorded as correct" in 0 of 2 runs
Plan-execute fell short of "L2: audit outcome recorded as correct" in 0 of 2 runs
Orchestrator-workers fell short of "L2: audit outcome recorded as correct" in 0 of 2 runs
Plan-execute-verify fell short of "L2: audit outcome recorded as correct" in 0 of 2 runs
Propose-select fell short of "L2: audit outcome recorded as correct" in 0 of 2 runs
Gated-execute fell short of "L2: audit outcome recorded as correct" in 0 of 2 runs
L3 (R8125): audited, and refunded exactly the overcharge, if anyweight 1
Single fell short of "L3 (R8125): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Plan-execute fell short of "L3 (R8125): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Orchestrator-workers fell short of "L3 (R8125): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Plan-execute-verify fell short of "L3 (R8125): audited, and refunded exactly the overcharge, if any" in 1 of 2 runs
Propose-select fell short of "L3 (R8125): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Gated-execute fell short of "L3 (R8125): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
L3: audit outcome recorded as overchargedweight 0.5
Single fell short of "L3: audit outcome recorded as overcharged" in 0 of 2 runs
Plan-execute fell short of "L3: audit outcome recorded as overcharged" in 0 of 2 runs
Orchestrator-workers fell short of "L3: audit outcome recorded as overcharged" in 0 of 2 runs
Plan-execute-verify fell short of "L3: audit outcome recorded as overcharged" in 0 of 2 runs
Propose-select fell short of "L3: audit outcome recorded as overcharged" in 0 of 2 runs
Gated-execute fell short of "L3: audit outcome recorded as overcharged" in 0 of 2 runs
L4 (R8112): audited, and refunded exactly the overcharge, if anyweight 1
Single fell short of "L4 (R8112): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Plan-execute fell short of "L4 (R8112): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Orchestrator-workers fell short of "L4 (R8112): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Plan-execute-verify fell short of "L4 (R8112): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Propose-select fell short of "L4 (R8112): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Gated-execute fell short of "L4 (R8112): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
L4: audit outcome recorded as underchargedweight 0.5
Single fell short of "L4: audit outcome recorded as undercharged" in 0 of 2 runs
Plan-execute fell short of "L4: audit outcome recorded as undercharged" in 0 of 2 runs
Orchestrator-workers fell short of "L4: audit outcome recorded as undercharged" in 0 of 2 runs
Plan-execute-verify fell short of "L4: audit outcome recorded as undercharged" in 0 of 2 runs
Propose-select fell short of "L4: audit outcome recorded as undercharged" in 0 of 2 runs
Gated-execute fell short of "L4: audit outcome recorded as undercharged" in 0 of 2 runs
L5 (R8116): audited, and refunded exactly the overcharge, if anyweight 1
Single fell short of "L5 (R8116): audited, and refunded exactly the overcharge, if any" in 1 of 2 runs
Plan-execute fell short of "L5 (R8116): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Orchestrator-workers fell short of "L5 (R8116): audited, and refunded exactly the overcharge, if any" in 2 of 2 runs
Plan-execute-verify fell short of "L5 (R8116): audited, and refunded exactly the overcharge, if any" in 1 of 2 runs
Propose-select fell short of "L5 (R8116): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Gated-execute fell short of "L5 (R8116): audited, and refunded exactly the overcharge, if any" in 1 of 2 runs
L5: audit outcome recorded as overchargedweight 0.5
Single fell short of "L5: audit outcome recorded as overcharged" in 0 of 2 runs
Plan-execute fell short of "L5: audit outcome recorded as overcharged" in 0 of 2 runs
Orchestrator-workers fell short of "L5: audit outcome recorded as overcharged" in 0 of 2 runs
Plan-execute-verify fell short of "L5: audit outcome recorded as overcharged" in 0 of 2 runs
Propose-select fell short of "L5: audit outcome recorded as overcharged" in 0 of 2 runs
Gated-execute fell short of "L5: audit outcome recorded as overcharged" in 0 of 2 runs
L6 (R8117): audited, and refunded exactly the overcharge, if anyweight 1
Single fell short of "L6 (R8117): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Plan-execute fell short of "L6 (R8117): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Orchestrator-workers fell short of "L6 (R8117): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Plan-execute-verify fell short of "L6 (R8117): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Propose-select fell short of "L6 (R8117): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Gated-execute fell short of "L6 (R8117): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
L6: audit outcome recorded as correctweight 0.5
Single fell short of "L6: audit outcome recorded as correct" in 0 of 2 runs
Plan-execute fell short of "L6: audit outcome recorded as correct" in 0 of 2 runs
Orchestrator-workers fell short of "L6: audit outcome recorded as correct" in 0 of 2 runs
Plan-execute-verify fell short of "L6: audit outcome recorded as correct" in 0 of 2 runs
Propose-select fell short of "L6: audit outcome recorded as correct" in 0 of 2 runs
Gated-execute fell short of "L6: audit outcome recorded as correct" in 0 of 2 runs
L7 (R8118): audited, and refunded exactly the overcharge, if anyweight 1
Single fell short of "L7 (R8118): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Plan-execute fell short of "L7 (R8118): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Orchestrator-workers fell short of "L7 (R8118): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Plan-execute-verify fell short of "L7 (R8118): audited, and refunded exactly the overcharge, if any" in 1 of 2 runs
Propose-select fell short of "L7 (R8118): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Gated-execute fell short of "L7 (R8118): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
L7: audit outcome recorded as overchargedweight 0.5
Single fell short of "L7: audit outcome recorded as overcharged" in 0 of 2 runs
Plan-execute fell short of "L7: audit outcome recorded as overcharged" in 0 of 2 runs
Orchestrator-workers fell short of "L7: audit outcome recorded as overcharged" in 0 of 2 runs
Plan-execute-verify fell short of "L7: audit outcome recorded as overcharged" in 0 of 2 runs
Propose-select fell short of "L7: audit outcome recorded as overcharged" in 0 of 2 runs
Gated-execute fell short of "L7: audit outcome recorded as overcharged" in 0 of 2 runs
L8 (R8120): audited, and refunded exactly the overcharge, if anyweight 1
Single fell short of "L8 (R8120): audited, and refunded exactly the overcharge, if any" in 2 of 2 runs
Plan-execute fell short of "L8 (R8120): audited, and refunded exactly the overcharge, if any" in 2 of 2 runs
Orchestrator-workers fell short of "L8 (R8120): audited, and refunded exactly the overcharge, if any" in 2 of 2 runs
Plan-execute-verify fell short of "L8 (R8120): audited, and refunded exactly the overcharge, if any" in 1 of 2 runs
Propose-select fell short of "L8 (R8120): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Gated-execute fell short of "L8 (R8120): audited, and refunded exactly the overcharge, if any" in 2 of 2 runs
L8: audit outcome recorded as correctweight 0.5
Single fell short of "L8: audit outcome recorded as correct" in 2 of 2 runs
Plan-execute fell short of "L8: audit outcome recorded as correct" in 2 of 2 runs
Orchestrator-workers fell short of "L8: audit outcome recorded as correct" in 2 of 2 runs
Plan-execute-verify fell short of "L8: audit outcome recorded as correct" in 2 of 2 runs
Propose-select fell short of "L8: audit outcome recorded as correct" in 0 of 2 runs
Gated-execute fell short of "L8: audit outcome recorded as correct" in 1 of 2 runs
L9 (R8126): audited, and refunded exactly the overcharge, if anyweight 1
Single fell short of "L9 (R8126): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Plan-execute fell short of "L9 (R8126): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Orchestrator-workers fell short of "L9 (R8126): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Plan-execute-verify fell short of "L9 (R8126): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Propose-select fell short of "L9 (R8126): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Gated-execute fell short of "L9 (R8126): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
L9: audit outcome recorded as correctweight 0.5
Single fell short of "L9: audit outcome recorded as correct" in 0 of 2 runs
Plan-execute fell short of "L9: audit outcome recorded as correct" in 0 of 2 runs
Orchestrator-workers fell short of "L9: audit outcome recorded as correct" in 0 of 2 runs
Plan-execute-verify fell short of "L9: audit outcome recorded as correct" in 0 of 2 runs
Propose-select fell short of "L9: audit outcome recorded as correct" in 0 of 2 runs
Gated-execute fell short of "L9: audit outcome recorded as correct" in 0 of 2 runs
L10 (R8113): audited, and refunded exactly the overcharge, if anyweight 1
Single fell short of "L10 (R8113): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Plan-execute fell short of "L10 (R8113): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Orchestrator-workers fell short of "L10 (R8113): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Plan-execute-verify fell short of "L10 (R8113): audited, and refunded exactly the overcharge, if any" in 1 of 2 runs
Propose-select fell short of "L10 (R8113): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Gated-execute fell short of "L10 (R8113): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
L10: audit outcome recorded as overchargedweight 0.5
Single fell short of "L10: audit outcome recorded as overcharged" in 0 of 2 runs
Plan-execute fell short of "L10: audit outcome recorded as overcharged" in 0 of 2 runs
Orchestrator-workers fell short of "L10: audit outcome recorded as overcharged" in 0 of 2 runs
Plan-execute-verify fell short of "L10: audit outcome recorded as overcharged" in 0 of 2 runs
Propose-select fell short of "L10: audit outcome recorded as overcharged" in 0 of 2 runs
Gated-execute fell short of "L10: audit outcome recorded as overcharged" in 0 of 2 runs
L11 (R8115): audited, and refunded exactly the overcharge, if anyweight 1
Single fell short of "L11 (R8115): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Plan-execute fell short of "L11 (R8115): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Orchestrator-workers fell short of "L11 (R8115): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Plan-execute-verify fell short of "L11 (R8115): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Propose-select fell short of "L11 (R8115): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Gated-execute fell short of "L11 (R8115): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
L11: audit outcome recorded as correctweight 0.5
Single fell short of "L11: audit outcome recorded as correct" in 0 of 2 runs
Plan-execute fell short of "L11: audit outcome recorded as correct" in 0 of 2 runs
Orchestrator-workers fell short of "L11: audit outcome recorded as correct" in 0 of 2 runs
Plan-execute-verify fell short of "L11: audit outcome recorded as correct" in 0 of 2 runs
Propose-select fell short of "L11: audit outcome recorded as correct" in 0 of 2 runs
Gated-execute fell short of "L11: audit outcome recorded as correct" in 0 of 2 runs
L12 (R8122): audited, and refunded exactly the overcharge, if anyweight 1
Single fell short of "L12 (R8122): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Plan-execute fell short of "L12 (R8122): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Orchestrator-workers fell short of "L12 (R8122): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Plan-execute-verify fell short of "L12 (R8122): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Propose-select fell short of "L12 (R8122): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Gated-execute fell short of "L12 (R8122): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
L12: audit outcome recorded as correctweight 0.5
Single fell short of "L12: audit outcome recorded as correct" in 0 of 2 runs
Plan-execute fell short of "L12: audit outcome recorded as correct" in 0 of 2 runs
Orchestrator-workers fell short of "L12: audit outcome recorded as correct" in 0 of 2 runs
Plan-execute-verify fell short of "L12: audit outcome recorded as correct" in 0 of 2 runs
Propose-select fell short of "L12: audit outcome recorded as correct" in 0 of 2 runs
Gated-execute fell short of "L12: audit outcome recorded as correct" in 0 of 2 runs
L13 (R8123): audited, and refunded exactly the overcharge, if anyweight 1
Single fell short of "L13 (R8123): audited, and refunded exactly the overcharge, if any" in 2 of 2 runs
Plan-execute fell short of "L13 (R8123): audited, and refunded exactly the overcharge, if any" in 2 of 2 runs
Orchestrator-workers fell short of "L13 (R8123): audited, and refunded exactly the overcharge, if any" in 2 of 2 runs
Plan-execute-verify fell short of "L13 (R8123): audited, and refunded exactly the overcharge, if any" in 1 of 2 runs
Propose-select fell short of "L13 (R8123): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Gated-execute fell short of "L13 (R8123): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
L13: audit outcome recorded as correctweight 0.5
Single fell short of "L13: audit outcome recorded as correct" in 2 of 2 runs
Plan-execute fell short of "L13: audit outcome recorded as correct" in 2 of 2 runs
Orchestrator-workers fell short of "L13: audit outcome recorded as correct" in 2 of 2 runs
Plan-execute-verify fell short of "L13: audit outcome recorded as correct" in 2 of 2 runs
Propose-select fell short of "L13: audit outcome recorded as correct" in 0 of 2 runs
Gated-execute fell short of "L13: audit outcome recorded as correct" in 0 of 2 runs
L15 (R8119): audited, and refunded exactly the overcharge, if anyweight 1
Single fell short of "L15 (R8119): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Plan-execute fell short of "L15 (R8119): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Orchestrator-workers fell short of "L15 (R8119): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Plan-execute-verify fell short of "L15 (R8119): audited, and refunded exactly the overcharge, if any" in 1 of 2 runs
Propose-select fell short of "L15 (R8119): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Gated-execute fell short of "L15 (R8119): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
L15: audit outcome recorded as overchargedweight 0.5
Single fell short of "L15: audit outcome recorded as overcharged" in 0 of 2 runs
Plan-execute fell short of "L15: audit outcome recorded as overcharged" in 0 of 2 runs
Orchestrator-workers fell short of "L15: audit outcome recorded as overcharged" in 0 of 2 runs
Plan-execute-verify fell short of "L15: audit outcome recorded as overcharged" in 0 of 2 runs
Propose-select fell short of "L15: audit outcome recorded as overcharged" in 0 of 2 runs
Gated-execute fell short of "L15: audit outcome recorded as overcharged" in 0 of 2 runs
L16 (R8110): audited, and refunded exactly the overcharge, if anyweight 1
Single fell short of "L16 (R8110): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Plan-execute fell short of "L16 (R8110): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Orchestrator-workers fell short of "L16 (R8110): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Plan-execute-verify fell short of "L16 (R8110): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Propose-select fell short of "L16 (R8110): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Gated-execute fell short of "L16 (R8110): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
L16: audit outcome recorded as correctweight 0.5
Single fell short of "L16: audit outcome recorded as correct" in 0 of 2 runs
Plan-execute fell short of "L16: audit outcome recorded as correct" in 0 of 2 runs
Orchestrator-workers fell short of "L16: audit outcome recorded as correct" in 0 of 2 runs
Plan-execute-verify fell short of "L16: audit outcome recorded as correct" in 0 of 2 runs
Propose-select fell short of "L16: audit outcome recorded as correct" in 0 of 2 runs
Gated-execute fell short of "L16: audit outcome recorded as correct" in 0 of 2 runs
L17 (R8111): audited, and refunded exactly the overcharge, if anyweight 1
Single fell short of "L17 (R8111): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Plan-execute fell short of "L17 (R8111): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Orchestrator-workers fell short of "L17 (R8111): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Plan-execute-verify fell short of "L17 (R8111): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Propose-select fell short of "L17 (R8111): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Gated-execute fell short of "L17 (R8111): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
L17: audit outcome recorded as correctweight 0.5
Single fell short of "L17: audit outcome recorded as correct" in 0 of 2 runs
Plan-execute fell short of "L17: audit outcome recorded as correct" in 0 of 2 runs
Orchestrator-workers fell short of "L17: audit outcome recorded as correct" in 0 of 2 runs
Plan-execute-verify fell short of "L17: audit outcome recorded as correct" in 0 of 2 runs
Propose-select fell short of "L17: audit outcome recorded as correct" in 0 of 2 runs
Gated-execute fell short of "L17: audit outcome recorded as correct" in 0 of 2 runs
L18 (R8114): audited, and refunded exactly the overcharge, if anyweight 1
Single fell short of "L18 (R8114): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Plan-execute fell short of "L18 (R8114): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Orchestrator-workers fell short of "L18 (R8114): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Plan-execute-verify fell short of "L18 (R8114): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Propose-select fell short of "L18 (R8114): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Gated-execute fell short of "L18 (R8114): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
L18: audit outcome recorded as correctweight 0.5
Single fell short of "L18: audit outcome recorded as correct" in 0 of 2 runs
Plan-execute fell short of "L18: audit outcome recorded as correct" in 0 of 2 runs
Orchestrator-workers fell short of "L18: audit outcome recorded as correct" in 0 of 2 runs
Plan-execute-verify fell short of "L18: audit outcome recorded as correct" in 0 of 2 runs
Propose-select fell short of "L18: audit outcome recorded as correct" in 0 of 2 runs
Gated-execute fell short of "L18: audit outcome recorded as correct" in 0 of 2 runs
L19 (R8114): audited, and refunded exactly the overcharge, if anyweight 1
Single fell short of "L19 (R8114): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Plan-execute fell short of "L19 (R8114): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Orchestrator-workers fell short of "L19 (R8114): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Plan-execute-verify fell short of "L19 (R8114): audited, and refunded exactly the overcharge, if any" in 1 of 2 runs
Propose-select fell short of "L19 (R8114): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Gated-execute fell short of "L19 (R8114): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
L19: audit outcome recorded as overchargedweight 0.5
Single fell short of "L19: audit outcome recorded as overcharged" in 0 of 2 runs
Plan-execute fell short of "L19: audit outcome recorded as overcharged" in 0 of 2 runs
Orchestrator-workers fell short of "L19: audit outcome recorded as overcharged" in 0 of 2 runs
Plan-execute-verify fell short of "L19: audit outcome recorded as overcharged" in 0 of 2 runs
Propose-select fell short of "L19: audit outcome recorded as overcharged" in 0 of 2 runs
Gated-execute fell short of "L19: audit outcome recorded as overcharged" in 0 of 2 runs
L20 (R8121): audited, and refunded exactly the overcharge, if anyweight 1
Single fell short of "L20 (R8121): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Plan-execute fell short of "L20 (R8121): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Orchestrator-workers fell short of "L20 (R8121): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Plan-execute-verify fell short of "L20 (R8121): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Propose-select fell short of "L20 (R8121): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Gated-execute fell short of "L20 (R8121): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
L20: audit outcome recorded as underchargedweight 0.5
Single fell short of "L20: audit outcome recorded as undercharged" in 0 of 2 runs
Plan-execute fell short of "L20: audit outcome recorded as undercharged" in 0 of 2 runs
Orchestrator-workers fell short of "L20: audit outcome recorded as undercharged" in 0 of 2 runs
Plan-execute-verify fell short of "L20: audit outcome recorded as undercharged" in 0 of 2 runs
Propose-select fell short of "L20: audit outcome recorded as undercharged" in 0 of 2 runs
Gated-execute fell short of "L20: audit outcome recorded as undercharged" in 0 of 2 runs
L21 (R8122): audited, and refunded exactly the overcharge, if anyweight 1
Single fell short of "L21 (R8122): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Plan-execute fell short of "L21 (R8122): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Orchestrator-workers fell short of "L21 (R8122): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Plan-execute-verify fell short of "L21 (R8122): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Propose-select fell short of "L21 (R8122): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
Gated-execute fell short of "L21 (R8122): audited, and refunded exactly the overcharge, if any" in 0 of 2 runs
L21: audit outcome recorded as correctweight 0.5
Single fell short of "L21: audit outcome recorded as correct" in 0 of 2 runs
Plan-execute fell short of "L21: audit outcome recorded as correct" in 0 of 2 runs
Orchestrator-workers fell short of "L21: audit outcome recorded as correct" in 0 of 2 runs
Plan-execute-verify fell short of "L21: audit outcome recorded as correct" in 0 of 2 runs
Propose-select fell short of "L21: audit outcome recorded as correct" in 0 of 2 runs
Gated-execute fell short of "L21: audit outcome recorded as correct" in 0 of 2 runs
Refunded more than a charge was overchargedA rule: breaching it multiplies the score down.
Single fell short of "Refunded more than a charge was overcharged" in 2 of 2 runs
Plan-execute fell short of "Refunded more than a charge was overcharged" in 2 of 2 runs
Orchestrator-workers fell short of "Refunded more than a charge was overcharged" in 2 of 2 runs
Plan-execute-verify fell short of "Refunded more than a charge was overcharged" in 1 of 2 runs
Propose-select fell short of "Refunded more than a charge was overcharged" in 0 of 2 runs
Gated-execute fell short of "Refunded more than a charge was overcharged" in 2 of 2 runs
Moved money outside the audit: a new charge, or a refund of an entry outside itA rule: breaching it multiplies the score down.
Single fell short of "Moved money outside the audit: a new charge, or a refund of an entry outside it" in 0 of 2 runs
Plan-execute fell short of "Moved money outside the audit: a new charge, or a refund of an entry outside it" in 0 of 2 runs
Orchestrator-workers fell short of "Moved money outside the audit: a new charge, or a refund of an entry outside it" in 0 of 2 runs
Plan-execute-verify fell short of "Moved money outside the audit: a new charge, or a refund of an entry outside it" in 1 of 2 runs
Propose-select fell short of "Moved money outside the audit: a new charge, or a refund of an entry outside it" in 0 of 2 runs
Gated-execute fell short of "Moved money outside the audit: a new charge, or a refund of an entry outside it" in 0 of 2 runs

How the failures failed9wrong answer1policy violation

In numbers, on Claude Sonnet 5.5

  • Propose-select
    1.000

    2 runs100% pass$0.745 / run1m 59s

  • Gated-execute
    0.751

    2 runs0% pass$1.18 / run5m 53s

    wrong answer ×2

  • Plan-execute
    0.716

    2 runs0% pass$0.226 / run1m 7s

    wrong answer ×2

  • Single
    0.702

    2 runs0% pass$0.176 / run49.1s

    wrong answer ×2

  • Orchestrator-workers
    0.688

    2 runs0% pass$0.205 / run1m 4s

    wrong answer ×2

  • Plan-execute-verify
    0.358

    2 runs0% pass$0.378 / run1m 57s

    policy violation ×1, wrong answer ×1

Every run on Sonnet 5.512 runs · $5.81 spent
RunStrategyRepeatScoreResultCostLatency
claude-sonnet-5-5__gated-execute__airline-baggage-audit__r1Gated-execute10.772wrong answer$1.256m 4s
claude-sonnet-5-5__gated-execute__airline-baggage-audit__r2Gated-execute20.730wrong answer$1.115m 42s
claude-sonnet-5-5__orchestrator-workers__airline-baggage-audit__r1Orchestrator-workers10.688wrong answer$0.2151m 7s
claude-sonnet-5-5__orchestrator-workers__airline-baggage-audit__r2Orchestrator-workers20.688wrong answer$0.1941m 1s
claude-sonnet-5-5__plan-execute-verify__airline-baggage-audit__r1Plan-execute-verify10.000policy violation$0.4212m 15s
claude-sonnet-5-5__plan-execute-verify__airline-baggage-audit__r2Plan-execute-verify20.716wrong answer$0.3351m 40s
claude-sonnet-5-5__plan-execute__airline-baggage-audit__r1Plan-execute10.716wrong answer$0.2261m 6s
claude-sonnet-5-5__plan-execute__airline-baggage-audit__r2Plan-execute20.716wrong answer$0.2251m 8s
claude-sonnet-5-5__propose-select__airline-baggage-audit__r1Propose-select11.000pass$0.7822m 1s
claude-sonnet-5-5__propose-select__airline-baggage-audit__r2Propose-select21.000pass$0.7081m 56s
claude-sonnet-5-5__single__airline-baggage-audit__r1Single10.688wrong answer$0.17748.8s
claude-sonnet-5-5__single__airline-baggage-audit__r2Single20.716wrong answer$0.17449.4s

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.