RunSonnet 5.5repeat 2
Single on Share two short deliveries out among backorders
claude-sonnet-5-5__single__ecommerce-backorder-allocation__r2
Score
0.697
- Cost
- $0.223
- Latency
- 54.1s
- setup 0.0s, grading 0.0s
- Turns
- 68
- Tool calls
- 67
- Subagents
- 0
- Tokens in / out
- 205k / 8.5k
- 178,113 read from cache
Did the design run?
- Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
- No API failures: heldharness.apiNo model call failed at the API.
- Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
- No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
- Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
- No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
- Builtin tools: heldharness.builtin-toolsA check recorded with the run.
- Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
- No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
- Each call was shown the right tools: heldsingle.toolsThe agent was shown exactly the task’s tools, none missing and none extra.
Phases
Drawn from the phase records: timing and cost per call, no tool calls or messages.
| Phase | Duration | Cost | Turns | Tools | Model |
|---|---|---|---|---|---|
| Main | 54.1s | $0.223 | 68 | 67 | claude-sonnet-5-5 |
The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.
Runs you can replay
- A gate that blocked the right moveGated-execute · Haiku 4.5
- Three proposals beat one agentPropose-select · Haiku 4.5
- A verifier that earned its keepPlan-execute-verify · Sonnet 5.5
- A revision that did not helpPlan-execute-verify · Haiku 4.5
- The control, missing a stepSingle · Haiku 4.5
Record this cell yourself: npm run bench -- run --strategies single --tasks ecommerce-backorder-allocation --models claude-sonnet-5-5 --repeats 1
Score breakdown
- passO-5201 allocated from the delivery1.00weight 0.05
L1=pending
- passO-5201: customer told once that the order is allocated1.00weight 0.01
- passO-5202 allocated from the delivery1.00weight 0.05
L1=pending L2=pending
- passO-5202: customer told once that the order is allocated1.00weight 0.01
- failO-5203 allocated from the delivery0.00weight 0.05
L1=backordered@2026-08-05
- passO-5203: customer told once that the order is allocated1.00weight 0.01
- passO-5204 allocated from the delivery1.00weight 0.05
L1=pending
- passO-5204: customer told once that the order is allocated1.00weight 0.01
- passO-5206 left waiting, with a promise date1.00weight 0.05
L1=backordered@2026-08-14 L2=backordered@2026-08-14 L3=pending
- passO-5206 waiting lines promised for the date the in-transit supply gives them1.00weight 0.03
expected L1@2026-08-14, L2@2026-08-14
- passO-5206: customer told once, with the promised date1.00weight 0.01
- passO-5207 allocated from the delivery1.00weight 0.05
L1=pending L2=pending
- passO-5207: customer told once that the order is allocated1.00weight 0.01
- passO-5208 allocated from the delivery1.00weight 0.05
L1=pending
- passO-5208: customer told once that the order is allocated1.00weight 0.01
- passO-5209 allocated from the delivery1.00weight 0.05
L1=pending
- passO-5209: customer told once that the order is allocated1.00weight 0.01
- passO-5210 allocated from the delivery1.00weight 0.05
L1=pending L2=pending
- passO-5210: customer told once that the order is allocated1.00weight 0.01
- passO-5211 allocated from the delivery1.00weight 0.05
L1=pending L2=shipped
- passO-5211: customer told once that the order is allocated1.00weight 0.01
- failO-5212 allocated from the delivery0.00weight 0.05
L1=backordered@2026-08-05
- passO-5212: customer told once that the order is allocated1.00weight 0.01
- passO-5213 left waiting, with a promise date1.00weight 0.05
L1=backordered@2026-08-14
- passO-5213 waiting lines promised for the date the in-transit supply gives them1.00weight 0.03
expected L1@2026-08-14
- passO-5213: customer told once, with the promised date1.00weight 0.01
- passO-5214 left waiting, with a promise date1.00weight 0.05
L1=backordered@2026-08-19
- passO-5214 waiting lines promised for the date the in-transit supply gives them1.00weight 0.03
expected L1@2026-08-19
- passO-5214: customer told once, with the promised date1.00weight 0.01
- passO-5215 allocated from the delivery1.00weight 0.05
L1=pending
- passO-5215: customer told once that the order is allocated1.00weight 0.01
- failBoth deliveries were allocated in full0.00weight 0.03
P-DOC stock=0, P-MON stock=3
Penalties
- policy violationAn order waited while lower-priority orders took the units it needed×0.8
O-5203 waited for P-MON while lower-priority orders took 3
Scorer metadata
- tool calls
- 67
- failed tool calls
- 0
- distinct tools
- 9
Final answer
I worked the backorders for P-DOC and P-MON. Ten lines are allocated, five are still waiting with promise dates, and one order was left alone. All 14 affected customers have been notified.
One thing differed from the task. PO-91 (4 docks) was marked received partway through, so I had 4 more docks on hand than the 7 from PO-88. I allocated those too under the same priority rules. I then left PO-91 out of the promise-date supply.
Allocated
Laptop docks (11 units):
- O-5204 L1, 2 units.
- O-5207 L1, 1 unit.
- O-5202 L2, 1 unit.
- O-5209 L1, 2 units.
- O-5210 L1, 1 unit.
- O-5201 L1, 2 units.
- O-5211 L1, 1 unit.
- O-5215 L1, 1 unit.
Monitors (4 units):
- O-5207 L2, 1 unit.
- O-5202 L1, 1 unit.
- O-5208 L1, 2 units.
- O-5207 is ship-complete. Both its lines could be allocated at once, so it went ahead.
Still waiting, with promise dates
- O-5206 is ship-complete and also has a USB-C Hub line with no stock. I promised both its lines for 2026-08-14. The dock would come from PO-95 on 08-14, and the hub from PO-90 on 08-06. The later date applies.
- O-5213 L1 (docks) is promised for 2026-08-14, from PO-95.
- O-5203 L1 and O-5212 L1 (monitors) are promised for 2026-08-05, from PO-93.
- O-5214 L1 (monitor) is promised for 2026-08-19, from PO-97.
Left alone: O-5205 is on risk hold. I did not allocate it, set a promise date, or contact the customer.
Notices: Each customer got one notice, either that their items are allocated and ready to ship, or the promise date for the waiting items.
Provenance of the numbers on this page
- run set
- scaffold-or-scale-v1
- models
- Haiku 4.5, Sonnet 5.5, Opus 5.5
- effort
- high
- repeats
- 2
- fixtures
- replay
- runs
- 900 / 900
- strategy cost
- $354.44
- judge cost
- $9.19
- wall clock
- 9h 48m
- git
- fce5807b2d
Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.