ScaffoldBench

tool-use · ecommerce

Cancel one line of a multi-line order

mediumpartial-cancellationrefundinventory
Task id
ecommerce-medium
Turn limit
18
Runs on Opus 5.5
12
Pass rate
100%

Who solved it

On Opus 5.5, every architecture averages 1.000: this task does not separate them. The same task, every architecture, every model: one square per run.

1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count

Point at a square or tab to it to read its run. Each square opens its run.

passedpartial creditfailedexcluded: integrity check failedexcluded: API errorOne square per run; each opens its run.

The prompt every architecture was handed

Marcus Ade (customer C-200) has emailed about order O-2003. He added the wireless mouse by mistake and wants that item, and only that item, cancelled and refunded to his card. The rest of the order should still ship as planned.

The domain policy

Appended to the system prompt for every architecture, so none of them gets a rules advantage.

Back-office handbook. You act for the store. These rules bind you even when a customer asks for something else. Scope - Change only what the request covers. Do not cancel, refund, dispatch, restock or edit any other order, line, customer or product. Orders and addresses - A line that has shipped cannot be cancelled; shipped goods can only be returned. - The shipping address of an order that has shipped is never changed: the parcel is in transit and the shipping record must stay accurate. Cancellations - A line that never shipped is a cancellation, not a return. It is refunded to the original payment method (method=card) for what the customer paid for it. - Cancelled goods never left the warehouse: put the full quantity of every cancelled line that had stock set aside for it back into stock. A backordered line has none set aside. Returns and refunds - The return policy (get_return_policy) sets the return window and how a return is paid out once the window has closed. - A refund for a line never exceeds what the customer actually paid for that line. - Returned goods re-enter stock only when the warehouse receives them. Issuing a refund does not put anything back on the shelf. - A store-credit refund is complete only when it is recorded against the order (refund_order) and the same amount is added to the customer balance (apply_store_credit).

Where it was lost

On Opus 5.5, no architecture fell short of any graded item. A score says an architecture lost points. This says which ones: for each graded item, the runs that earned less than full credit, out of the runs that count.

Graded items on Opus 5.5runs that fell short, of runs that count
Graded item
Single
Plan-execute
Orchestrator-workers
Plan-execute-verify
Propose-select
Gated-execute
The mouse line is cancelled and refunded to the card for exactly what was paidweight 3
Single fell short of "The mouse line is cancelled and refunded to the card for exactly what was paid" in 0 of 2 runs
Plan-execute fell short of "The mouse line is cancelled and refunded to the card for exactly what was paid" in 0 of 2 runs
Orchestrator-workers fell short of "The mouse line is cancelled and refunded to the card for exactly what was paid" in 0 of 2 runs
Plan-execute-verify fell short of "The mouse line is cancelled and refunded to the card for exactly what was paid" in 0 of 2 runs
Propose-select fell short of "The mouse line is cancelled and refunded to the card for exactly what was paid" in 0 of 2 runs
Gated-execute fell short of "The mouse line is cancelled and refunded to the card for exactly what was paid" in 0 of 2 runs
Exactly the cancelled quantity went back into stockweight 2
Single fell short of "Exactly the cancelled quantity went back into stock" in 0 of 2 runs
Plan-execute fell short of "Exactly the cancelled quantity went back into stock" in 0 of 2 runs
Orchestrator-workers fell short of "Exactly the cancelled quantity went back into stock" in 0 of 2 runs
Plan-execute-verify fell short of "Exactly the cancelled quantity went back into stock" in 0 of 2 runs
Propose-select fell short of "Exactly the cancelled quantity went back into stock" in 0 of 2 runs
Gated-execute fell short of "Exactly the cancelled quantity went back into stock" in 0 of 2 runs
The other lines of the order are still on their wayweight 2
Single fell short of "The other lines of the order are still on their way" in 0 of 2 runs
Plan-execute fell short of "The other lines of the order are still on their way" in 0 of 2 runs
Orchestrator-workers fell short of "The other lines of the order are still on their way" in 0 of 2 runs
Plan-execute-verify fell short of "The other lines of the order are still on their way" in 0 of 2 runs
Propose-select fell short of "The other lines of the order are still on their way" in 0 of 2 runs
Gated-execute fell short of "The other lines of the order are still on their way" in 0 of 2 runs
Read the order before changing itweight 1
Single fell short of "Read the order before changing it" in 0 of 2 runs
Plan-execute fell short of "Read the order before changing it" in 0 of 2 runs
Orchestrator-workers fell short of "Read the order before changing it" in 0 of 2 runs
Plan-execute-verify fell short of "Read the order before changing it" in 0 of 2 runs
Propose-select fell short of "Read the order before changing it" in 0 of 2 runs
Gated-execute fell short of "Read the order before changing it" in 0 of 2 runs

In numbers, on Claude Opus 5.5

  • Gated-execute
    1.000

    2 runs100% pass$0.118 / run36.9s

  • Orchestrator-workers
    1.000

    2 runs100% pass$0.102 / run32.6s

  • Plan-execute
    1.000

    2 runs100% pass$0.085 / run25.0s

  • Plan-execute-verify
    1.000

    2 runs100% pass$0.113 / run32.5s

  • Propose-select
    1.000

    2 runs100% pass$0.266 / run44.8s

  • Single
    1.000

    2 runs100% pass$0.046 / run14.7s

Every run on Opus 5.512 runs · $1.46 spent
RunStrategyRepeatScoreResultCostLatency
claude-opus-5-5__gated-execute__ecommerce-medium__r1Gated-execute11.000pass$0.11736.5s
claude-opus-5-5__gated-execute__ecommerce-medium__r2Gated-execute21.000pass$0.11937.2s
claude-opus-5-5__orchestrator-workers__ecommerce-medium__r1Orchestrator-workers11.000pass$0.10432.8s
claude-opus-5-5__orchestrator-workers__ecommerce-medium__r2Orchestrator-workers21.000pass$0.10132.5s
claude-opus-5-5__plan-execute-verify__ecommerce-medium__r1Plan-execute-verify11.000pass$0.11433.3s
claude-opus-5-5__plan-execute-verify__ecommerce-medium__r2Plan-execute-verify21.000pass$0.11331.7s
claude-opus-5-5__plan-execute__ecommerce-medium__r1Plan-execute11.000pass$0.08926.3s
claude-opus-5-5__plan-execute__ecommerce-medium__r2Plan-execute21.000pass$0.08123.7s
claude-opus-5-5__propose-select__ecommerce-medium__r1Propose-select11.000pass$0.27146.8s
claude-opus-5-5__propose-select__ecommerce-medium__r2Propose-select21.000pass$0.26242.8s
claude-opus-5-5__single__ecommerce-medium__r1Single11.000pass$0.04514.8s
claude-opus-5-5__single__ecommerce-medium__r2Single21.000pass$0.04714.6s

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.