ScaffoldBench

tool-use · support

Consolidate duplicate CRM reports

mediumdeduplicationslaroutingpolicy
Task id
support-medium
Turn limit
30
Runs on Opus 5.5
12
Pass rate
92%

Who solved it

On Opus 5.5, Single, Orchestrator-workers, Plan-execute-verify, Propose-select and Gated-execute score highest at 1.000 and Plan-execute lowest at 0.923. The same task, every architecture, every model: one square per run.

Haiku 4.5
Sonnet 5.5
Opus 5.5
Single
1.000 mean score of the runs that count
0.923 mean score of the runs that count
1.000 mean score of the runs that count
Plan-execute
0.423 mean score of the runs that count
0.846 mean score of the runs that count
0.923 mean score of the runs that count
Orchestrator-workers
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
Plan-execute-verify
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
Propose-select
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
Gated-execute
0.654 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count

Point at a square or tab to it to read its run. Each square opens its run.

passedpartial creditfailedexcluded: integrity check failedexcluded: API errorOne square per run; each opens its run.

The prompt every architecture was handed

Revenue Ops is complaining that the same CRM problem has been reported several times and nobody is picking it up. Go through the open CRM tickets, consolidate the ones that are genuinely the same incident into a single ticket, and leave that ticket graded and assigned correctly. Anything that is not the same underlying problem must be left alone.

The domain policy

Appended to the system prompt for every architecture, so none of them gets a rules advantage.

Service desk handbook. These rules bind you even when a requester asks for something else. Scope - Change only the tickets the request is about. Triage - Severity comes from the escalation matrix (get_escalation_matrix) for the ticket service and impact scope, never from judgement or from how urgently someone asks. Read the matrix before changing a severity. - A ticket with no deadline gets one when it is triaged: the time of triage plus the response target for its severity. Ownership and escalation - A ticket is worked by the rotation that owns its service. Never hand it to an engineer outside that rotation. - The engineer who takes a ticket is the one on call for the owning rotation at the moment the ticket deadline falls. - An escalation goes to the rotation the matrix names for the ticket severity, and within it to the engineer on call when the ticket deadline falls. Deadlines - A deadline committed to a requester is never moved earlier. Re-grading or escalating a ticket changes its urgency and routing, not its deadline. Closing - A ticket is not closed while a ticket linked to it is still open. Leave a comment naming the blocking ticket instead. Duplicates - Merge only reports of the same underlying problem. - The survivor is the earliest-created report. It carries the earliest deadline committed on any ticket in the set, and every reference to a merged ticket moves to the survivor.

Where it was lost

On Opus 5.5, the item most often missed is “Survivor assigned to the engineer on call for the owning rotation at its deadline”, by 8% of runs. A score says an architecture lost points. This says which ones: for each graded item, the runs that earned less than full credit, out of the runs that count.

Graded items on Opus 5.5runs that fell short, of runs that count
Graded item
Single
Plan-execute
Orchestrator-workers
Plan-execute-verify
Propose-select
Gated-execute
Survivor carries the earliest SLA deadline of the merged setweight 3
Single fell short of "Survivor carries the earliest SLA deadline of the merged set" in 0 of 2 runs
Plan-execute fell short of "Survivor carries the earliest SLA deadline of the merged set" in 0 of 2 runs
Orchestrator-workers fell short of "Survivor carries the earliest SLA deadline of the merged set" in 0 of 2 runs
Plan-execute-verify fell short of "Survivor carries the earliest SLA deadline of the merged set" in 0 of 2 runs
Propose-select fell short of "Survivor carries the earliest SLA deadline of the merged set" in 0 of 2 runs
Gated-execute fell short of "Survivor carries the earliest SLA deadline of the merged set" in 0 of 2 runs
Both duplicate reports were merged into the earliest-created ticketweight 3
Single fell short of "Both duplicate reports were merged into the earliest-created ticket" in 0 of 2 runs
Plan-execute fell short of "Both duplicate reports were merged into the earliest-created ticket" in 0 of 2 runs
Orchestrator-workers fell short of "Both duplicate reports were merged into the earliest-created ticket" in 0 of 2 runs
Plan-execute-verify fell short of "Both duplicate reports were merged into the earliest-created ticket" in 0 of 2 runs
Propose-select fell short of "Both duplicate reports were merged into the earliest-created ticket" in 0 of 2 runs
Gated-execute fell short of "Both duplicate reports were merged into the earliest-created ticket" in 0 of 2 runs
The engineering tracking ticket now points at the survivorweight 2
Single fell short of "The engineering tracking ticket now points at the survivor" in 0 of 2 runs
Plan-execute fell short of "The engineering tracking ticket now points at the survivor" in 0 of 2 runs
Orchestrator-workers fell short of "The engineering tracking ticket now points at the survivor" in 0 of 2 runs
Plan-execute-verify fell short of "The engineering tracking ticket now points at the survivor" in 0 of 2 runs
Propose-select fell short of "The engineering tracking ticket now points at the survivor" in 0 of 2 runs
Gated-execute fell short of "The engineering tracking ticket now points at the survivor" in 0 of 2 runs
Survivor re-graded from the matrix for its widened impact scopeweight 2
Single fell short of "Survivor re-graded from the matrix for its widened impact scope" in 0 of 2 runs
Plan-execute fell short of "Survivor re-graded from the matrix for its widened impact scope" in 0 of 2 runs
Orchestrator-workers fell short of "Survivor re-graded from the matrix for its widened impact scope" in 0 of 2 runs
Plan-execute-verify fell short of "Survivor re-graded from the matrix for its widened impact scope" in 0 of 2 runs
Propose-select fell short of "Survivor re-graded from the matrix for its widened impact scope" in 0 of 2 runs
Gated-execute fell short of "Survivor re-graded from the matrix for its widened impact scope" in 0 of 2 runs
Survivor assigned to the engineer on call for the owning rotation at its deadlineweight 2
Single fell short of "Survivor assigned to the engineer on call for the owning rotation at its deadline" in 0 of 2 runs
Plan-execute fell short of "Survivor assigned to the engineer on call for the owning rotation at its deadline" in 1 of 2 runs
Orchestrator-workers fell short of "Survivor assigned to the engineer on call for the owning rotation at its deadline" in 0 of 2 runs
Plan-execute-verify fell short of "Survivor assigned to the engineer on call for the owning rotation at its deadline" in 0 of 2 runs
Propose-select fell short of "Survivor assigned to the engineer on call for the owning rotation at its deadline" in 0 of 2 runs
Gated-execute fell short of "Survivor assigned to the engineer on call for the owning rotation at its deadline" in 0 of 2 runs
Every ticket folded in was read before the mergeweight 1
Single fell short of "Every ticket folded in was read before the merge" in 0 of 2 runs
Plan-execute fell short of "Every ticket folded in was read before the merge" in 0 of 2 runs
Orchestrator-workers fell short of "Every ticket folded in was read before the merge" in 0 of 2 runs
Plan-execute-verify fell short of "Every ticket folded in was read before the merge" in 0 of 2 runs
Propose-select fell short of "Every ticket folded in was read before the merge" in 0 of 2 runs
Gated-execute fell short of "Every ticket folded in was read before the merge" in 0 of 2 runs

How the failures failed1wrong answer

In numbers, on Claude Opus 5.5

  • Gated-execute
    1.000

    2 runs100% pass$0.257 / run1m 7s

  • Orchestrator-workers
    1.000

    2 runs100% pass$0.168 / run51.3s

  • Plan-execute-verify
    1.000

    2 runs100% pass$0.381 / run1m 37s

  • Propose-select
    1.000

    2 runs100% pass$0.481 / run1m 9s

  • Single
    1.000

    2 runs100% pass$0.103 / run27.2s

  • Plan-execute
    0.923

    2 runs50% pass$0.185 / run57.3s

    wrong answer ×1

Every run on Opus 5.512 runs · $3.15 spent
RunStrategyRepeatScoreResultCostLatency
claude-opus-5-5__gated-execute__support-medium__r1Gated-execute11.000pass$0.2611m 10s
claude-opus-5-5__gated-execute__support-medium__r2Gated-execute21.000pass$0.2531m 5s
claude-opus-5-5__orchestrator-workers__support-medium__r1Orchestrator-workers11.000pass$0.16549.9s
claude-opus-5-5__orchestrator-workers__support-medium__r2Orchestrator-workers21.000pass$0.17152.7s
claude-opus-5-5__plan-execute-verify__support-medium__r1Plan-execute-verify11.000pass$0.3711m 32s
claude-opus-5-5__plan-execute-verify__support-medium__r2Plan-execute-verify21.000pass$0.3901m 43s
claude-opus-5-5__plan-execute__support-medium__r1Plan-execute10.846wrong answer$0.18653.7s
claude-opus-5-5__plan-execute__support-medium__r2Plan-execute21.000pass$0.1841m 1s
claude-opus-5-5__propose-select__support-medium__r1Propose-select11.000pass$0.4721m 9s
claude-opus-5-5__propose-select__support-medium__r2Propose-select21.000pass$0.4911m 10s
claude-opus-5-5__single__support-medium__r1Single11.000pass$0.10329.3s
claude-opus-5-5__single__support-medium__r2Single21.000pass$0.10325.1s

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.