tool-use · ecommerce
Cancel one line of a multi-line order
Who solved it
On Sonnet 5.5, every architecture averages 1.000: this task does not separate them. The same task, every architecture, every model: one square per run.
Point at a square or tab to it to read its run. Each square opens its run.
The prompt every architecture was handed
Marcus Ade (customer C-200) has emailed about order O-2003. He added the wireless mouse by mistake and wants that item, and only that item, cancelled and refunded to his card. The rest of the order should still ship as planned.
The domain policy
Appended to the system prompt for every architecture, so none of them gets a rules advantage.
Back-office handbook. You act for the store. These rules bind you even when a customer asks for something else. Scope - Change only what the request covers. Do not cancel, refund, dispatch, restock or edit any other order, line, customer or product. Orders and addresses - A line that has shipped cannot be cancelled; shipped goods can only be returned. - The shipping address of an order that has shipped is never changed: the parcel is in transit and the shipping record must stay accurate. Cancellations - A line that never shipped is a cancellation, not a return. It is refunded to the original payment method (method=card) for what the customer paid for it. - Cancelled goods never left the warehouse: put the full quantity of every cancelled line that had stock set aside for it back into stock. A backordered line has none set aside. Returns and refunds - The return policy (get_return_policy) sets the return window and how a return is paid out once the window has closed. - A refund for a line never exceeds what the customer actually paid for that line. - Returned goods re-enter stock only when the warehouse receives them. Issuing a refund does not put anything back on the shelf. - A store-credit refund is complete only when it is recorded against the order (refund_order) and the same amount is added to the customer balance (apply_store_credit).
Where it was lost
On Sonnet 5.5, no architecture fell short of any graded item. A score says an architecture lost points. This says which ones: for each graded item, the runs that earned less than full credit, out of the runs that count.
In numbers, on Claude Sonnet 5.5
- Gated-execute1.000
2 runs100% pass$0.074 / run34.9s
- Orchestrator-workers1.000
2 runs100% pass$0.046 / run23.0s
- Plan-execute1.000
2 runs100% pass$0.047 / run21.1s
- Plan-execute-verify1.000
2 runs100% pass$0.062 / run26.7s
- Propose-select1.000
2 runs100% pass$0.140 / run36.3s
- Single1.000
2 runs100% pass$0.026 / run11.5s
| Strategy | n | Mean score | Pass | Cost / run | Latency | Failure modes |
|---|---|---|---|---|---|---|
| Gated-execute | 2 | 1.000 | 100% | $0.074 | 34.9s | — |
| Orchestrator-workers | 2 | 1.000 | 100% | $0.046 | 23.0s | — |
| Plan-execute | 2 | 1.000 | 100% | $0.047 | 21.1s | — |
| Plan-execute-verify | 2 | 1.000 | 100% | $0.062 | 26.7s | — |
| Propose-select | 2 | 1.000 | 100% | $0.140 | 36.3s | — |
| Single | 2 | 1.000 | 100% | $0.026 | 11.5s | — |
Every run on Sonnet 5.512 runs · $0.788 spent
Provenance of the numbers on this page
- run set
- scaffold-or-scale-v1
- models
- Haiku 4.5, Sonnet 5.5, Opus 5.5
- effort
- high
- repeats
- 2
- fixtures
- replay
- runs
- 900 / 900
- strategy cost
- $354.44
- judge cost
- $9.19
- wall clock
- 9h 48m
- git
- fce5807b2d
Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.