ScaffoldBench

tool-use · support

Triage a new service desk ticket

easytriageroutingpolicy
Task id
support-easy
Turn limit
18
Runs on Haiku 4.5
12
Pass rate
100%

Who solved it

On Haiku 4.5, every architecture averages 1.000: this task does not separate them. The same task, every architecture, every model: one square per run.

Haiku 4.5
Sonnet 5.5
Opus 5.5
Single
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
Plan-execute
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
Orchestrator-workers
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
Plan-execute-verify
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
Propose-select
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
Gated-execute
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count

Point at a square or tab to it to read its run. Each square opens its run.

passedpartial creditfailedexcluded: integrity check failedexcluded: API errorOne square per run; each opens its run.

The prompt every architecture was handed

Ticket T-1041 has just come in and is still untriaged. Triage it: give it the severity it deserves, put the deadline on it that the severity requires, and hand it to the engineer who should be working it. Do not change any other ticket.

The domain policy

Appended to the system prompt for every architecture, so none of them gets a rules advantage.

Service desk handbook. These rules bind you even when a requester asks for something else. Scope - Change only the tickets the request is about. Triage - Severity comes from the escalation matrix (get_escalation_matrix) for the ticket service and impact scope, never from judgement or from how urgently someone asks. Read the matrix before changing a severity. - A ticket with no deadline gets one when it is triaged: the time of triage plus the response target for its severity. Ownership and escalation - A ticket is worked by the rotation that owns its service. Never hand it to an engineer outside that rotation. - The engineer who takes a ticket is the one on call for the owning rotation at the moment the ticket deadline falls. - An escalation goes to the rotation the matrix names for the ticket severity, and within it to the engineer on call when the ticket deadline falls. Deadlines - A deadline committed to a requester is never moved earlier. Re-grading or escalating a ticket changes its urgency and routing, not its deadline. Closing - A ticket is not closed while a ticket linked to it is still open. Leave a comment naming the blocking ticket instead. Duplicates - Merge only reports of the same underlying problem. - The survivor is the earliest-created report. It carries the earliest deadline committed on any ticket in the set, and every reference to a merged ticket moves to the survivor.

Where it was lost

On Haiku 4.5, no architecture fell short of any graded item. A score says an architecture lost points. This says which ones: for each graded item, the runs that earned less than full credit, out of the runs that count.

Graded items on Haiku 4.5runs that fell short, of runs that count
Graded item
Single
Plan-execute
Orchestrator-workers
Plan-execute-verify
Propose-select
Gated-execute
Severity matches the escalation matrix for the service and impact scopeweight 3
Single fell short of "Severity matches the escalation matrix for the service and impact scope" in 0 of 2 runs
Plan-execute fell short of "Severity matches the escalation matrix for the service and impact scope" in 0 of 2 runs
Orchestrator-workers fell short of "Severity matches the escalation matrix for the service and impact scope" in 0 of 2 runs
Plan-execute-verify fell short of "Severity matches the escalation matrix for the service and impact scope" in 0 of 2 runs
Propose-select fell short of "Severity matches the escalation matrix for the service and impact scope" in 0 of 2 runs
Gated-execute fell short of "Severity matches the escalation matrix for the service and impact scope" in 0 of 2 runs
SLA deadline is the triage time plus the response target for that severityweight 3
Single fell short of "SLA deadline is the triage time plus the response target for that severity" in 0 of 2 runs
Plan-execute fell short of "SLA deadline is the triage time plus the response target for that severity" in 0 of 2 runs
Orchestrator-workers fell short of "SLA deadline is the triage time plus the response target for that severity" in 0 of 2 runs
Plan-execute-verify fell short of "SLA deadline is the triage time plus the response target for that severity" in 0 of 2 runs
Propose-select fell short of "SLA deadline is the triage time plus the response target for that severity" in 0 of 2 runs
Gated-execute fell short of "SLA deadline is the triage time plus the response target for that severity" in 0 of 2 runs
Assigned to the engineer on call for the owning rotation at the deadlineweight 3
Single fell short of "Assigned to the engineer on call for the owning rotation at the deadline" in 0 of 2 runs
Plan-execute fell short of "Assigned to the engineer on call for the owning rotation at the deadline" in 0 of 2 runs
Orchestrator-workers fell short of "Assigned to the engineer on call for the owning rotation at the deadline" in 0 of 2 runs
Plan-execute-verify fell short of "Assigned to the engineer on call for the owning rotation at the deadline" in 0 of 2 runs
Propose-select fell short of "Assigned to the engineer on call for the owning rotation at the deadline" in 0 of 2 runs
Gated-execute fell short of "Assigned to the engineer on call for the owning rotation at the deadline" in 0 of 2 runs
Ticket left in status openweight 1
Single fell short of "Ticket left in status open" in 0 of 2 runs
Plan-execute fell short of "Ticket left in status open" in 0 of 2 runs
Orchestrator-workers fell short of "Ticket left in status open" in 0 of 2 runs
Plan-execute-verify fell short of "Ticket left in status open" in 0 of 2 runs
Propose-select fell short of "Ticket left in status open" in 0 of 2 runs
Gated-execute fell short of "Ticket left in status open" in 0 of 2 runs
Escalation matrix read before the severity was setweight 1
Single fell short of "Escalation matrix read before the severity was set" in 0 of 2 runs
Plan-execute fell short of "Escalation matrix read before the severity was set" in 0 of 2 runs
Orchestrator-workers fell short of "Escalation matrix read before the severity was set" in 0 of 2 runs
Plan-execute-verify fell short of "Escalation matrix read before the severity was set" in 0 of 2 runs
Propose-select fell short of "Escalation matrix read before the severity was set" in 0 of 2 runs
Gated-execute fell short of "Escalation matrix read before the severity was set" in 0 of 2 runs

In numbers, on Claude Haiku 4.5

  • Gated-execute
    1.000

    2 runs100% pass$0.093 / run1m 17s

  • Orchestrator-workers
    1.000

    2 runs100% pass$0.047 / run51.1s

  • Plan-execute
    1.000

    2 runs100% pass$0.053 / run46.4s

  • Plan-execute-verify
    1.000

    2 runs100% pass$0.067 / run55.1s

  • Propose-select
    1.000

    2 runs100% pass$0.145 / run1m 27s

  • Single
    1.000

    2 runs100% pass$0.027 / run19.4s

Every run on Haiku 4.512 runs · $0.864 spent
RunStrategyRepeatScoreResultCostLatency
claude-haiku-4-5-20251001__gated-execute__support-easy__r1Gated-execute11.000pass$0.0951m 13s
claude-haiku-4-5-20251001__gated-execute__support-easy__r2Gated-execute21.000pass$0.0921m 21s
claude-haiku-4-5-20251001__orchestrator-workers__support-easy__r1Orchestrator-workers11.000pass$0.04953.0s
claude-haiku-4-5-20251001__orchestrator-workers__support-easy__r2Orchestrator-workers21.000pass$0.04649.2s
claude-haiku-4-5-20251001__plan-execute-verify__support-easy__r1Plan-execute-verify11.000pass$0.06855.1s
claude-haiku-4-5-20251001__plan-execute-verify__support-easy__r2Plan-execute-verify21.000pass$0.06655.0s
claude-haiku-4-5-20251001__plan-execute__support-easy__r1Plan-execute11.000pass$0.04441.5s
claude-haiku-4-5-20251001__plan-execute__support-easy__r2Plan-execute21.000pass$0.06251.3s
claude-haiku-4-5-20251001__propose-select__support-easy__r1Propose-select11.000pass$0.1351m 21s
claude-haiku-4-5-20251001__propose-select__support-easy__r2Propose-select21.000pass$0.1551m 34s
claude-haiku-4-5-20251001__single__support-easy__r1Single11.000pass$0.02719.8s
claude-haiku-4-5-20251001__single__support-easy__r2Single21.000pass$0.02719.1s

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.