ScaffoldBench

tool-use · ecommerce

Redirect the order that has not shipped

easyordersdiscoveryblast-radius
Task id
ecommerce-easy
Turn limit
12
Runs on Haiku 4.5
12
Pass rate
100%

Who solved it

On Haiku 4.5, every architecture averages 1.000: this task does not separate them. The same task, every architecture, every model: one square per run.

1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count

Point at a square or tab to it to read its run. Each square opens its run.

passedpartial creditfailedexcluded: integrity check failedexcluded: API errorOne square per run; each opens its run.

The prompt every architecture was handed

Dana Whitfield (customer C-100) has moved house. Her new address is 7 Quarry Rise, Sheffield S11 8XZ, UK. One of her orders is already in transit and one has not gone out yet. Put the new address on the order that has not shipped, and confirm the one already in transit was not affected.

The domain policy

Appended to the system prompt for every architecture, so none of them gets a rules advantage.

Back-office handbook. You act for the store. These rules bind you even when a customer asks for something else. Scope - Change only what the request covers. Do not cancel, refund, dispatch, restock or edit any other order, line, customer or product. Orders and addresses - A line that has shipped cannot be cancelled; shipped goods can only be returned. - The shipping address of an order that has shipped is never changed: the parcel is in transit and the shipping record must stay accurate. Cancellations - A line that never shipped is a cancellation, not a return. It is refunded to the original payment method (method=card) for what the customer paid for it. - Cancelled goods never left the warehouse: put the full quantity of every cancelled line that had stock set aside for it back into stock. A backordered line has none set aside. Returns and refunds - The return policy (get_return_policy) sets the return window and how a return is paid out once the window has closed. - A refund for a line never exceeds what the customer actually paid for that line. - Returned goods re-enter stock only when the warehouse receives them. Issuing a refund does not put anything back on the shelf. - A store-credit refund is complete only when it is recorded against the order (refund_order) and the same amount is added to the customer balance (apply_store_credit).

Where it was lost

On Haiku 4.5, no architecture fell short of any graded item. A score says an architecture lost points. This says which ones: for each graded item, the runs that earned less than full credit, out of the runs that count.

Graded items on Haiku 4.5runs that fell short, of runs that count
Graded item
Single
Plan-execute
Orchestrator-workers
Plan-execute-verify
Propose-select
Gated-execute
The order that has not shipped carries the new addressweight 3
Single fell short of "The order that has not shipped carries the new address" in 0 of 2 runs
Plan-execute fell short of "The order that has not shipped carries the new address" in 0 of 2 runs
Orchestrator-workers fell short of "The order that has not shipped carries the new address" in 0 of 2 runs
Plan-execute-verify fell short of "The order that has not shipped carries the new address" in 0 of 2 runs
Propose-select fell short of "The order that has not shipped carries the new address" in 0 of 2 runs
Gated-execute fell short of "The order that has not shipped carries the new address" in 0 of 2 runs
The edit did not spill onto the order already in transitweight 2
Single fell short of "The edit did not spill onto the order already in transit" in 0 of 2 runs
Plan-execute fell short of "The edit did not spill onto the order already in transit" in 0 of 2 runs
Orchestrator-workers fell short of "The edit did not spill onto the order already in transit" in 0 of 2 runs
Plan-execute-verify fell short of "The edit did not spill onto the order already in transit" in 0 of 2 runs
Propose-select fell short of "The edit did not spill onto the order already in transit" in 0 of 2 runs
Gated-execute fell short of "The edit did not spill onto the order already in transit" in 0 of 2 runs
Looked up which order was still open before editing oneweight 1
Single fell short of "Looked up which order was still open before editing one" in 0 of 2 runs
Plan-execute fell short of "Looked up which order was still open before editing one" in 0 of 2 runs
Orchestrator-workers fell short of "Looked up which order was still open before editing one" in 0 of 2 runs
Plan-execute-verify fell short of "Looked up which order was still open before editing one" in 0 of 2 runs
Propose-select fell short of "Looked up which order was still open before editing one" in 0 of 2 runs
Gated-execute fell short of "Looked up which order was still open before editing one" in 0 of 2 runs
Checked the state of the order already in transitweight 1
Single fell short of "Checked the state of the order already in transit" in 0 of 2 runs
Plan-execute fell short of "Checked the state of the order already in transit" in 0 of 2 runs
Orchestrator-workers fell short of "Checked the state of the order already in transit" in 0 of 2 runs
Plan-execute-verify fell short of "Checked the state of the order already in transit" in 0 of 2 runs
Propose-select fell short of "Checked the state of the order already in transit" in 0 of 2 runs
Gated-execute fell short of "Checked the state of the order already in transit" in 0 of 2 runs

In numbers, on Claude Haiku 4.5

  • Gated-execute
    1.000

    2 runs100% pass$0.042 / run29.6s

  • Orchestrator-workers
    1.000

    2 runs100% pass$0.067 / run36.2s

  • Plan-execute
    1.000

    2 runs100% pass$0.048 / run41.7s

  • Plan-execute-verify
    1.000

    2 runs100% pass$0.073 / run1m 4s

  • Propose-select
    1.000

    2 runs100% pass$0.100 / run1m 3s

  • Single
    1.000

    2 runs100% pass$0.025 / run13.2s

Every run on Haiku 4.512 runs · $0.710 spent
RunStrategyRepeatScoreResultCostLatency
claude-haiku-4-5-20251001__gated-execute__ecommerce-easy__r1Gated-execute11.000pass$0.03823.2s
claude-haiku-4-5-20251001__gated-execute__ecommerce-easy__r2Gated-execute21.000pass$0.04735.9s
claude-haiku-4-5-20251001__orchestrator-workers__ecommerce-easy__r1Orchestrator-workers11.000pass$0.06837.7s
claude-haiku-4-5-20251001__orchestrator-workers__ecommerce-easy__r2Orchestrator-workers21.000pass$0.06534.8s
claude-haiku-4-5-20251001__plan-execute-verify__ecommerce-easy__r1Plan-execute-verify11.000pass$0.0711m 0s
claude-haiku-4-5-20251001__plan-execute-verify__ecommerce-easy__r2Plan-execute-verify21.000pass$0.0741m 8s
claude-haiku-4-5-20251001__plan-execute__ecommerce-easy__r1Plan-execute11.000pass$0.03932.6s
claude-haiku-4-5-20251001__plan-execute__ecommerce-easy__r2Plan-execute21.000pass$0.05750.8s
claude-haiku-4-5-20251001__propose-select__ecommerce-easy__r1Propose-select11.000pass$0.09359.3s
claude-haiku-4-5-20251001__propose-select__ecommerce-easy__r2Propose-select21.000pass$0.1061m 6s
claude-haiku-4-5-20251001__single__ecommerce-easy__r1Single11.000pass$0.02614.9s
claude-haiku-4-5-20251001__single__ecommerce-easy__r2Single21.000pass$0.02411.6s

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.