tool-use · ecommerce
Redirect the order that has not shipped
Who solved it
On Sonnet 5.5, every architecture averages 1.000: this task does not separate them. The same task, every architecture, every model: one square per run.
Point at a square or tab to it to read its run. Each square opens its run.
The prompt every architecture was handed
Dana Whitfield (customer C-100) has moved house. Her new address is 7 Quarry Rise, Sheffield S11 8XZ, UK. One of her orders is already in transit and one has not gone out yet. Put the new address on the order that has not shipped, and confirm the one already in transit was not affected.
The domain policy
Appended to the system prompt for every architecture, so none of them gets a rules advantage.
Back-office handbook. You act for the store. These rules bind you even when a customer asks for something else. Scope - Change only what the request covers. Do not cancel, refund, dispatch, restock or edit any other order, line, customer or product. Orders and addresses - A line that has shipped cannot be cancelled; shipped goods can only be returned. - The shipping address of an order that has shipped is never changed: the parcel is in transit and the shipping record must stay accurate. Cancellations - A line that never shipped is a cancellation, not a return. It is refunded to the original payment method (method=card) for what the customer paid for it. - Cancelled goods never left the warehouse: put the full quantity of every cancelled line that had stock set aside for it back into stock. A backordered line has none set aside. Returns and refunds - The return policy (get_return_policy) sets the return window and how a return is paid out once the window has closed. - A refund for a line never exceeds what the customer actually paid for that line. - Returned goods re-enter stock only when the warehouse receives them. Issuing a refund does not put anything back on the shelf. - A store-credit refund is complete only when it is recorded against the order (refund_order) and the same amount is added to the customer balance (apply_store_credit).
Where it was lost
On Sonnet 5.5, no architecture fell short of any graded item. A score says an architecture lost points. This says which ones: for each graded item, the runs that earned less than full credit, out of the runs that count.
In numbers, on Claude Sonnet 5.5
- Gated-execute1.000
2 runs100% pass$0.042 / run15.7s
- Orchestrator-workers1.000
2 runs100% pass$0.063 / run24.0s
- Plan-execute1.000
2 runs100% pass$0.041 / run17.0s
- Plan-execute-verify1.000
2 runs100% pass$0.061 / run21.8s
- Propose-select1.000
2 runs100% pass$0.134 / run28.0s
- Single1.000
2 runs100% pass$0.033 / run9.1s
| Strategy | n | Mean score | Pass | Cost / run | Latency | Failure modes |
|---|---|---|---|---|---|---|
| Gated-execute | 2 | 1.000 | 100% | $0.042 | 15.7s | — |
| Orchestrator-workers | 2 | 1.000 | 100% | $0.063 | 24.0s | — |
| Plan-execute | 2 | 1.000 | 100% | $0.041 | 17.0s | — |
| Plan-execute-verify | 2 | 1.000 | 100% | $0.061 | 21.8s | — |
| Propose-select | 2 | 1.000 | 100% | $0.134 | 28.0s | — |
| Single | 2 | 1.000 | 100% | $0.033 | 9.1s | — |
Every run on Sonnet 5.512 runs · $0.749 spent
Provenance of the numbers on this page
- run set
- scaffold-or-scale-v1
- models
- Haiku 4.5, Sonnet 5.5, Opus 5.5
- effort
- high
- repeats
- 2
- fixtures
- replay
- runs
- 900 / 900
- strategy cost
- $354.44
- judge cost
- $9.19
- wall clock
- 9h 48m
- git
- fce5807b2d
Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.