ScaffoldBench

Replay

Watch a run, call by call.

A score is the end of a story. The replay is the story: which model calls ran, in parallel or not, which tools they called, what the sandbox guard refused, and what the run had spent at every moment.

Cancel a fare the customer wrongly believes is refundable to their card

The same task for 6 architectures · Haiku 4.5 · repeat 1 · run set scaffold-or-scale-v1

5 calls · 2m 21s · $0.199

0.0s / 2m 21sspent$0
Proposal 1 (policy-first): 0.0s to 27.5s, finished, 4 tool calls, $0.027
Proposal 2 (state-first): 0.0s to 26.9s, finished, 4 tool calls, $0.027
Proposal 3 (outcome-first): 0.0s to 39.4s, finished, 4 tool calls, $0.039
Select: 39.4s to 1m 50s, finished, 2 tool calls, $0.070
Execute: 1m 50s to 2m 21s, finished, 6 tool calls, $0.037

Events as they happened0 / 41

  1. Nothing has happened yet.

Every message, tool call and denial of this run is on its own page: claude-haiku-4-5-20251001__propose-select__airline-hard__r1

Run it yourself

Runs use whatever Claude credentials your shell has, so they spend tokens on that account. Results land in results/, and --update-latest makes the site render them.

  1. 01Install

    npm ci

  2. 02Run two architectures on one task, once each

    npm run bench -- run --strategies single,plan-execute --tasks airline-hard --repeats 1 --update-latest

  3. 03Open the site on your results

    npm run dev:web

Live, in the browser

This copy of the site is static, so it cannot run a model and will never ask for an API key. To try the streaming demo, serve the site in server mode on your machine with NEXT_PUBLIC_SCAFFOLDBENCH_LIVE=1 npm run dev:web; runs there are billed to the key you paste and to no other credentials.