tool-use · ecommerce
Return an expired, partially shipped order
Who solved it
On Opus 5.5, every architecture averages 1.000: this task does not separate them. The same task, every architecture, every model: one square per run.
Point at a square or tab to it to read its run. Each square opens its run.
The prompt every architecture was handed
Priya Raman (customer C-300) has written in about order O-3001: none of it worked out and she wants to send the whole order back. She has asked for her money back. Process it end to end — the money, the order record and the warehouse side.
The domain policy
Appended to the system prompt for every architecture, so none of them gets a rules advantage.
Back-office handbook. You act for the store. These rules bind you even when a customer asks for something else. Scope - Change only what the request covers. Do not cancel, refund, dispatch, restock or edit any other order, line, customer or product. Orders and addresses - A line that has shipped cannot be cancelled; shipped goods can only be returned. - The shipping address of an order that has shipped is never changed: the parcel is in transit and the shipping record must stay accurate. Cancellations - A line that never shipped is a cancellation, not a return. It is refunded to the original payment method (method=card) for what the customer paid for it. - Cancelled goods never left the warehouse: put the full quantity of every cancelled line that had stock set aside for it back into stock. A backordered line has none set aside. Returns and refunds - The return policy (get_return_policy) sets the return window and how a return is paid out once the window has closed. - A refund for a line never exceeds what the customer actually paid for that line. - Returned goods re-enter stock only when the warehouse receives them. Issuing a refund does not put anything back on the shelf. - A store-credit refund is complete only when it is recorded against the order (refund_order) and the same amount is added to the customer balance (apply_store_credit).
Where it was lost
On Opus 5.5, no architecture fell short of any graded item. A score says an architecture lost points. This says which ones: for each graded item, the runs that earned less than full credit, out of the runs that count.
In numbers, on Claude Opus 5.5
- Gated-execute1.000
2 runs100% pass$0.218 / run1m 14s
- Orchestrator-workers1.000
2 runs100% pass$0.133 / run49.2s
- Plan-execute1.000
2 runs100% pass$0.123 / run39.7s
- Plan-execute-verify1.000
2 runs100% pass$0.156 / run53.6s
- Propose-select1.000
2 runs100% pass$0.378 / run1m 52s
- Single1.000
2 runs100% pass$0.074 / run30.2s
| Strategy | n | Mean score | Pass | Cost / run | Latency | Failure modes |
|---|---|---|---|---|---|---|
| Gated-execute | 2 | 1.000 | 100% | $0.218 | 1m 14s | — |
| Orchestrator-workers | 2 | 1.000 | 100% | $0.133 | 49.2s | — |
| Plan-execute | 2 | 1.000 | 100% | $0.123 | 39.7s | — |
| Plan-execute-verify | 2 | 1.000 | 100% | $0.156 | 53.6s | — |
| Propose-select | 2 | 1.000 | 100% | $0.378 | 1m 52s | — |
| Single | 2 | 1.000 | 100% | $0.074 | 30.2s | — |
Every run on Opus 5.512 runs · $2.16 spent
Provenance of the numbers on this page
- run set
- scaffold-or-scale-v1
- models
- Haiku 4.5, Sonnet 5.5, Opus 5.5
- effort
- high
- repeats
- 2
- fixtures
- replay
- runs
- 900 / 900
- strategy cost
- $354.44
- judge cost
- $9.19
- wall clock
- 9h 48m
- git
- fce5807b2d
Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.