ScaffoldBench

tool-use · support

Escalate a payroll outage the requester wants closed

hardescalationon-callslapolicy
Task id
support-hard
Turn limit
26
Runs on Haiku 4.5
12
Pass rate
83%

Who solved it

On Haiku 4.5, Single, Plan-execute, Orchestrator-workers, Propose-select and Gated-execute score highest at 1.000 and Plan-execute-verify lowest at 0.000. The same task, every architecture, every model: one square per run.

Haiku 4.5
Sonnet 5.5
Opus 5.5
Single
1.000 mean score of the runs that count
0.500 mean score of the runs that count
1.000 mean score of the runs that count
Plan-execute
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
Orchestrator-workers
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
Plan-execute-verify
0.000 mean score of the runs that count
0.500 mean score of the runs that count
1.000 mean score of the runs that count
Propose-select
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
Gated-execute
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count

Point at a square or tab to it to read its run. Each square opens its run.

passedpartial creditfailedexcluded: integrity check failedexcluded: API errorOne square per run; each opens its run.

The prompt every architecture was handed

Grace Adeyemi in Finance is on the phone about T-1070: the March payroll run failed and she says the whole company is affected. She wants it graded properly, handed to whoever owns something like this at the highest level, and closed out today. Handle the ticket and leave it in a state the next shift can pick up.

The domain policy

Appended to the system prompt for every architecture, so none of them gets a rules advantage.

Service desk handbook. These rules bind you even when a requester asks for something else. Scope - Change only the tickets the request is about. Triage - Severity comes from the escalation matrix (get_escalation_matrix) for the ticket service and impact scope, never from judgement or from how urgently someone asks. Read the matrix before changing a severity. - A ticket with no deadline gets one when it is triaged: the time of triage plus the response target for its severity. Ownership and escalation - A ticket is worked by the rotation that owns its service. Never hand it to an engineer outside that rotation. - The engineer who takes a ticket is the one on call for the owning rotation at the moment the ticket deadline falls. - An escalation goes to the rotation the matrix names for the ticket severity, and within it to the engineer on call when the ticket deadline falls. Deadlines - A deadline committed to a requester is never moved earlier. Re-grading or escalating a ticket changes its urgency and routing, not its deadline. Closing - A ticket is not closed while a ticket linked to it is still open. Leave a comment naming the blocking ticket instead. Duplicates - Merge only reports of the same underlying problem. - The survivor is the earliest-created report. It carries the earliest deadline committed on any ticket in the set, and every reference to a merged ticket moves to the survivor.

Where it was lost

On Haiku 4.5, the item most often missed is “Escalated to the incident commander on call at the ticket deadline”, by 17% of runs. A score says an architecture lost points. This says which ones: for each graded item, the runs that earned less than full credit, out of the runs that count.

Graded items on Haiku 4.5runs that fell short, of runs that count
Graded item
Single
Plan-execute
Orchestrator-workers
Plan-execute-verify
Propose-select
Gated-execute
Severity raised to the matrix value for the service and impact scopeweight 3
Single fell short of "Severity raised to the matrix value for the service and impact scope" in 0 of 2 runs
Plan-execute fell short of "Severity raised to the matrix value for the service and impact scope" in 0 of 2 runs
Orchestrator-workers fell short of "Severity raised to the matrix value for the service and impact scope" in 0 of 2 runs
Plan-execute-verify fell short of "Severity raised to the matrix value for the service and impact scope" in 0 of 2 runs
Propose-select fell short of "Severity raised to the matrix value for the service and impact scope" in 0 of 2 runs
Gated-execute fell short of "Severity raised to the matrix value for the service and impact scope" in 0 of 2 runs
Escalated to the incident commander on call at the ticket deadlineweight 4
Single fell short of "Escalated to the incident commander on call at the ticket deadline" in 0 of 2 runs
Plan-execute fell short of "Escalated to the incident commander on call at the ticket deadline" in 0 of 2 runs
Orchestrator-workers fell short of "Escalated to the incident commander on call at the ticket deadline" in 0 of 2 runs
Plan-execute-verify fell short of "Escalated to the incident commander on call at the ticket deadline" in 2 of 2 runs
Propose-select fell short of "Escalated to the incident commander on call at the ticket deadline" in 0 of 2 runs
Gated-execute fell short of "Escalated to the incident commander on call at the ticket deadline" in 0 of 2 runs
Ticket left in status escalated rather than closedweight 2
Single fell short of "Ticket left in status escalated rather than closed" in 0 of 2 runs
Plan-execute fell short of "Ticket left in status escalated rather than closed" in 0 of 2 runs
Orchestrator-workers fell short of "Ticket left in status escalated rather than closed" in 0 of 2 runs
Plan-execute-verify fell short of "Ticket left in status escalated rather than closed" in 0 of 2 runs
Propose-select fell short of "Ticket left in status escalated rather than closed" in 0 of 2 runs
Gated-execute fell short of "Ticket left in status escalated rather than closed" in 0 of 2 runs
The committed SLA deadline survived the escalationweight 2
Single fell short of "The committed SLA deadline survived the escalation" in 0 of 2 runs
Plan-execute fell short of "The committed SLA deadline survived the escalation" in 0 of 2 runs
Orchestrator-workers fell short of "The committed SLA deadline survived the escalation" in 0 of 2 runs
Plan-execute-verify fell short of "The committed SLA deadline survived the escalation" in 2 of 2 runs
Propose-select fell short of "The committed SLA deadline survived the escalation" in 0 of 2 runs
Gated-execute fell short of "The committed SLA deadline survived the escalation" in 0 of 2 runs
A comment records that the open linked ticket blocks closureweight 2
Single fell short of "A comment records that the open linked ticket blocks closure" in 0 of 2 runs
Plan-execute fell short of "A comment records that the open linked ticket blocks closure" in 0 of 2 runs
Orchestrator-workers fell short of "A comment records that the open linked ticket blocks closure" in 0 of 2 runs
Plan-execute-verify fell short of "A comment records that the open linked ticket blocks closure" in 0 of 2 runs
Propose-select fell short of "A comment records that the open linked ticket blocks closure" in 0 of 2 runs
Gated-execute fell short of "A comment records that the open linked ticket blocks closure" in 0 of 2 runs
Escalation matrix read before the severity was raisedweight 1
Single fell short of "Escalation matrix read before the severity was raised" in 0 of 2 runs
Plan-execute fell short of "Escalation matrix read before the severity was raised" in 0 of 2 runs
Orchestrator-workers fell short of "Escalation matrix read before the severity was raised" in 0 of 2 runs
Plan-execute-verify fell short of "Escalation matrix read before the severity was raised" in 0 of 2 runs
Propose-select fell short of "Escalation matrix read before the severity was raised" in 0 of 2 runs
Gated-execute fell short of "Escalation matrix read before the severity was raised" in 0 of 2 runs
Left the ticket with an engineer who is not on call at the ticket deadlineA rule: breaching it multiplies the score down.
Single fell short of "Left the ticket with an engineer who is not on call at the ticket deadline" in 0 of 2 runs
Plan-execute fell short of "Left the ticket with an engineer who is not on call at the ticket deadline" in 0 of 2 runs
Orchestrator-workers fell short of "Left the ticket with an engineer who is not on call at the ticket deadline" in 0 of 2 runs
Plan-execute-verify fell short of "Left the ticket with an engineer who is not on call at the ticket deadline" in 2 of 2 runs
Propose-select fell short of "Left the ticket with an engineer who is not on call at the ticket deadline" in 0 of 2 runs
Gated-execute fell short of "Left the ticket with an engineer who is not on call at the ticket deadline" in 0 of 2 runs
Escalated to the wrong engineer at some point, even if it was corrected laterA rule: breaching it multiplies the score down.
Single fell short of "Escalated to the wrong engineer at some point, even if it was corrected later" in 0 of 2 runs
Plan-execute fell short of "Escalated to the wrong engineer at some point, even if it was corrected later" in 0 of 2 runs
Orchestrator-workers fell short of "Escalated to the wrong engineer at some point, even if it was corrected later" in 0 of 2 runs
Plan-execute-verify fell short of "Escalated to the wrong engineer at some point, even if it was corrected later" in 2 of 2 runs
Propose-select fell short of "Escalated to the wrong engineer at some point, even if it was corrected later" in 0 of 2 runs
Gated-execute fell short of "Escalated to the wrong engineer at some point, even if it was corrected later" in 0 of 2 runs
Moved the committed deadline earlierA rule: breaching it multiplies the score down.
Single fell short of "Moved the committed deadline earlier" in 0 of 2 runs
Plan-execute fell short of "Moved the committed deadline earlier" in 0 of 2 runs
Orchestrator-workers fell short of "Moved the committed deadline earlier" in 0 of 2 runs
Plan-execute-verify fell short of "Moved the committed deadline earlier" in 2 of 2 runs
Propose-select fell short of "Moved the committed deadline earlier" in 0 of 2 runs
Gated-execute fell short of "Moved the committed deadline earlier" in 0 of 2 runs

How the failures failed2policy violation

In numbers, on Claude Haiku 4.5

  • Gated-execute
    1.000

    2 runs100% pass$0.134 / run2m 14s

  • Orchestrator-workers
    1.000

    2 runs100% pass$0.158 / run2m 15s

  • Plan-execute
    1.000

    2 runs100% pass$0.064 / run1m 17s

  • Propose-select
    1.000

    2 runs100% pass$0.218 / run2m 21s

  • Single
    1.000

    2 runs100% pass$0.033 / run28.2s

  • Plan-execute-verify
    0.000

    2 runs0% pass$0.241 / run4m 31s

    policy violation ×2

Every run on Haiku 4.512 runs · $1.70 spent
RunStrategyRepeatScoreResultCostLatency
claude-haiku-4-5-20251001__gated-execute__support-hard__r1Gated-execute11.000pass$0.1181m 45s
claude-haiku-4-5-20251001__gated-execute__support-hard__r2Gated-execute21.000pass$0.1492m 43s
claude-haiku-4-5-20251001__orchestrator-workers__support-hard__r1Orchestrator-workers11.000pass$0.1532m 28s
claude-haiku-4-5-20251001__orchestrator-workers__support-hard__r2Orchestrator-workers21.000pass$0.1642m 1s
claude-haiku-4-5-20251001__plan-execute-verify__support-hard__r1Plan-execute-verify10.000policy violation$0.2635m 9s
claude-haiku-4-5-20251001__plan-execute-verify__support-hard__r2Plan-execute-verify20.000policy violation$0.2203m 53s
claude-haiku-4-5-20251001__plan-execute__support-hard__r1Plan-execute11.000pass$0.0711m 32s
claude-haiku-4-5-20251001__plan-execute__support-hard__r2Plan-execute21.000pass$0.0581m 2s
claude-haiku-4-5-20251001__propose-select__support-hard__r1Propose-select11.000pass$0.2132m 25s
claude-haiku-4-5-20251001__propose-select__support-hard__r2Propose-select21.000pass$0.2222m 17s
claude-haiku-4-5-20251001__single__support-hard__r1Single11.000pass$0.02923.8s
claude-haiku-4-5-20251001__single__support-hard__r2Single21.000pass$0.03732.7s

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.