ScaffoldBench

tool-use · ecommerce

Return an expired, partially shipped order

hardreturnsrefundpolicy-conflictinventory
Task id
ecommerce-hard
Turn limit
26
Runs on Haiku 4.5
12
Pass rate
75%

Who solved it

On Haiku 4.5, Single, Plan-execute, Plan-execute-verify and Propose-select score highest at 1.000 and Gated-execute lowest at 0.000. The same task, every architecture, every model: one square per run.

Haiku 4.5
Sonnet 5.5
Opus 5.5
Single
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
Plan-execute
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
Orchestrator-workers
0.705 mean score of the runs that count
0.909 mean score of the runs that count
1.000 mean score of the runs that count
Plan-execute-verify
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
Propose-select
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
Gated-execute
0.000 mean score of the runs that count
0.818 mean score of the runs that count
1.000 mean score of the runs that count

Point at a square or tab to it to read its run. Each square opens its run.

passedpartial creditfailedexcluded: integrity check failedexcluded: API errorOne square per run; each opens its run.

The prompt every architecture was handed

Priya Raman (customer C-300) has written in about order O-3001: none of it worked out and she wants to send the whole order back. She has asked for her money back. Process it end to end — the money, the order record and the warehouse side.

The domain policy

Appended to the system prompt for every architecture, so none of them gets a rules advantage.

Back-office handbook. You act for the store. These rules bind you even when a customer asks for something else. Scope - Change only what the request covers. Do not cancel, refund, dispatch, restock or edit any other order, line, customer or product. Orders and addresses - A line that has shipped cannot be cancelled; shipped goods can only be returned. - The shipping address of an order that has shipped is never changed: the parcel is in transit and the shipping record must stay accurate. Cancellations - A line that never shipped is a cancellation, not a return. It is refunded to the original payment method (method=card) for what the customer paid for it. - Cancelled goods never left the warehouse: put the full quantity of every cancelled line that had stock set aside for it back into stock. A backordered line has none set aside. Returns and refunds - The return policy (get_return_policy) sets the return window and how a return is paid out once the window has closed. - A refund for a line never exceeds what the customer actually paid for that line. - Returned goods re-enter stock only when the warehouse receives them. Issuing a refund does not put anything back on the shelf. - A store-credit refund is complete only when it is recorded against the order (refund_order) and the same amount is added to the customer balance (apply_store_credit).

Where it was lost

On Haiku 4.5, the item most often missed is “Cancelled units were restocked and returned units were not”, by 25% of runs. A score says an architecture lost points. This says which ones: for each graded item, the runs that earned less than full credit, out of the runs that count.

Graded items on Haiku 4.5runs that fell short, of runs that count
Graded item
Single
Plan-execute
Orchestrator-workers
Plan-execute-verify
Propose-select
Gated-execute
The two shipped lines were returned as store credit for exactly what they costweight 3
Single fell short of "The two shipped lines were returned as store credit for exactly what they cost" in 0 of 2 runs
Plan-execute fell short of "The two shipped lines were returned as store credit for exactly what they cost" in 0 of 2 runs
Orchestrator-workers fell short of "The two shipped lines were returned as store credit for exactly what they cost" in 0 of 2 runs
Plan-execute-verify fell short of "The two shipped lines were returned as store credit for exactly what they cost" in 0 of 2 runs
Propose-select fell short of "The two shipped lines were returned as store credit for exactly what they cost" in 0 of 2 runs
Gated-execute fell short of "The two shipped lines were returned as store credit for exactly what they cost" in 2 of 2 runs
The credit reached the customer balance, not just the ledgerweight 2
Single fell short of "The credit reached the customer balance, not just the ledger" in 0 of 2 runs
Plan-execute fell short of "The credit reached the customer balance, not just the ledger" in 0 of 2 runs
Orchestrator-workers fell short of "The credit reached the customer balance, not just the ledger" in 0 of 2 runs
Plan-execute-verify fell short of "The credit reached the customer balance, not just the ledger" in 0 of 2 runs
Propose-select fell short of "The credit reached the customer balance, not just the ledger" in 0 of 2 runs
Gated-execute fell short of "The credit reached the customer balance, not just the ledger" in 2 of 2 runs
The line that never shipped was cancelled and refunded to the cardweight 3
Single fell short of "The line that never shipped was cancelled and refunded to the card" in 0 of 2 runs
Plan-execute fell short of "The line that never shipped was cancelled and refunded to the card" in 0 of 2 runs
Orchestrator-workers fell short of "The line that never shipped was cancelled and refunded to the card" in 0 of 2 runs
Plan-execute-verify fell short of "The line that never shipped was cancelled and refunded to the card" in 0 of 2 runs
Propose-select fell short of "The line that never shipped was cancelled and refunded to the card" in 0 of 2 runs
Gated-execute fell short of "The line that never shipped was cancelled and refunded to the card" in 2 of 2 runs
Cancelled units were restocked and returned units were notweight 2
Single fell short of "Cancelled units were restocked and returned units were not" in 0 of 2 runs
Plan-execute fell short of "Cancelled units were restocked and returned units were not" in 0 of 2 runs
Orchestrator-workers fell short of "Cancelled units were restocked and returned units were not" in 1 of 2 runs
Plan-execute-verify fell short of "Cancelled units were restocked and returned units were not" in 0 of 2 runs
Propose-select fell short of "Cancelled units were restocked and returned units were not" in 0 of 2 runs
Gated-execute fell short of "Cancelled units were restocked and returned units were not" in 2 of 2 runs
Read the return policy before issuing a refundweight 1
Single fell short of "Read the return policy before issuing a refund" in 0 of 2 runs
Plan-execute fell short of "Read the return policy before issuing a refund" in 0 of 2 runs
Orchestrator-workers fell short of "Read the return policy before issuing a refund" in 0 of 2 runs
Plan-execute-verify fell short of "Read the return policy before issuing a refund" in 0 of 2 runs
Propose-select fell short of "Read the return policy before issuing a refund" in 0 of 2 runs
Gated-execute fell short of "Read the return policy before issuing a refund" in 2 of 2 runs
Put returned goods back into stock before the warehouse received themA rule: breaching it multiplies the score down.
Single fell short of "Put returned goods back into stock before the warehouse received them" in 0 of 2 runs
Plan-execute fell short of "Put returned goods back into stock before the warehouse received them" in 0 of 2 runs
Orchestrator-workers fell short of "Put returned goods back into stock before the warehouse received them" in 1 of 2 runs
Plan-execute-verify fell short of "Put returned goods back into stock before the warehouse received them" in 0 of 2 runs
Propose-select fell short of "Put returned goods back into stock before the warehouse received them" in 0 of 2 runs
Gated-execute fell short of "Put returned goods back into stock before the warehouse received them" in 0 of 2 runs

How the failures failed3wrong answer

In numbers, on Claude Haiku 4.5

  • Plan-execute
    1.000

    2 runs100% pass$0.067 / run1m 7s

  • Plan-execute-verify
    1.000

    2 runs100% pass$0.122 / run2m 27s

  • Propose-select
    1.000

    2 runs100% pass$0.152 / run1m 51s

  • Single
    1.000

    2 runs100% pass$0.037 / run30.6s

  • Orchestrator-workers
    0.705

    2 runs50% pass$0.154 / run1m 46s

    wrong answer ×1

  • Gated-execute
    0.000

    2 runs0% pass$2.74 / run36m 23s

    wrong answer ×2

Every run on Haiku 4.512 runs · $6.55 spent
RunStrategyRepeatScoreResultCostLatency
claude-haiku-4-5-20251001__gated-execute__ecommerce-hard__r1Gated-execute10.000wrong answer$3.5345m 22s
claude-haiku-4-5-20251001__gated-execute__ecommerce-hard__r2Gated-execute20.000wrong answer$1.9627m 25s
claude-haiku-4-5-20251001__orchestrator-workers__ecommerce-hard__r1Orchestrator-workers10.409wrong answer$0.1671m 57s
claude-haiku-4-5-20251001__orchestrator-workers__ecommerce-hard__r2Orchestrator-workers21.000pass$0.1411m 36s
claude-haiku-4-5-20251001__plan-execute-verify__ecommerce-hard__r1Plan-execute-verify11.000pass$0.1092m 12s
claude-haiku-4-5-20251001__plan-execute-verify__ecommerce-hard__r2Plan-execute-verify21.000pass$0.1362m 41s
claude-haiku-4-5-20251001__plan-execute__ecommerce-hard__r1Plan-execute11.000pass$0.0731m 15s
claude-haiku-4-5-20251001__plan-execute__ecommerce-hard__r2Plan-execute21.000pass$0.06059.8s
claude-haiku-4-5-20251001__propose-select__ecommerce-hard__r1Propose-select11.000pass$0.1441m 35s
claude-haiku-4-5-20251001__propose-select__ecommerce-hard__r2Propose-select21.000pass$0.1612m 8s
claude-haiku-4-5-20251001__single__ecommerce-hard__r1Single11.000pass$0.04032.3s
claude-haiku-4-5-20251001__single__ecommerce-hard__r2Single21.000pass$0.03528.8s

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.