tool-use · support
Consolidate duplicate CRM reports
Who solved it
On Haiku 4.5, Single, Orchestrator-workers, Plan-execute-verify and Propose-select score highest at 1.000 and Plan-execute lowest at 0.423. The same task, every architecture, every model: one square per run.
Point at a square or tab to it to read its run. Each square opens its run.
The prompt every architecture was handed
Revenue Ops is complaining that the same CRM problem has been reported several times and nobody is picking it up. Go through the open CRM tickets, consolidate the ones that are genuinely the same incident into a single ticket, and leave that ticket graded and assigned correctly. Anything that is not the same underlying problem must be left alone.
The domain policy
Appended to the system prompt for every architecture, so none of them gets a rules advantage.
Service desk handbook. These rules bind you even when a requester asks for something else. Scope - Change only the tickets the request is about. Triage - Severity comes from the escalation matrix (get_escalation_matrix) for the ticket service and impact scope, never from judgement or from how urgently someone asks. Read the matrix before changing a severity. - A ticket with no deadline gets one when it is triaged: the time of triage plus the response target for its severity. Ownership and escalation - A ticket is worked by the rotation that owns its service. Never hand it to an engineer outside that rotation. - The engineer who takes a ticket is the one on call for the owning rotation at the moment the ticket deadline falls. - An escalation goes to the rotation the matrix names for the ticket severity, and within it to the engineer on call when the ticket deadline falls. Deadlines - A deadline committed to a requester is never moved earlier. Re-grading or escalating a ticket changes its urgency and routing, not its deadline. Closing - A ticket is not closed while a ticket linked to it is still open. Leave a comment naming the blocking ticket instead. Duplicates - Merge only reports of the same underlying problem. - The survivor is the earliest-created report. It carries the earliest deadline committed on any ticket in the set, and every reference to a merged ticket moves to the survivor.
Where it was lost
On Haiku 4.5, the item most often missed is “Survivor assigned to the engineer on call for the owning rotation at its deadline”, by 25% of runs. A score says an architecture lost points. This says which ones: for each graded item, the runs that earned less than full credit, out of the runs that count.
How the failures failed2wrong answer1policy violation
In numbers, on Claude Haiku 4.5
- Orchestrator-workers1.000
2 runs100% pass$0.071 / run1m 9s
- Plan-execute-verify1.000
2 runs100% pass$0.229 / run4m 18s
- Propose-select1.000
2 runs100% pass$0.251 / run2m 40s
- Single1.000
2 runs100% pass$0.053 / run43.1s
- Gated-execute0.654
2 runs50% pass$0.816 / run14m 14s
wrong answer ×1
- Plan-execute0.423
2 runs0% pass$0.093 / run1m 38s
wrong answer ×1, policy violation ×1
| Strategy | n | Mean score | Pass | Cost / run | Latency | Failure modes |
|---|---|---|---|---|---|---|
| Orchestrator-workers | 2 | 1.000 | 100% | $0.071 | 1m 9s | — |
| Plan-execute-verify | 2 | 1.000 | 100% | $0.229 | 4m 18s | — |
| Propose-select | 2 | 1.000 | 100% | $0.251 | 2m 40s | — |
| Single | 2 | 1.000 | 100% | $0.053 | 43.1s | — |
| Gated-execute | 2 | 0.654 | 50% | $0.816 | 14m 14s | wrong answer ×1 |
| Plan-execute | 2 | 0.423 | 0% | $0.093 | 1m 38s | wrong answer ×1, policy violation ×1 |
Every run on Haiku 4.512 runs · $3.03 spent
Provenance of the numbers on this page
- run set
- scaffold-or-scale-v1
- models
- Haiku 4.5, Sonnet 5.5, Opus 5.5
- effort
- high
- repeats
- 2
- fixtures
- replay
- runs
- 900 / 900
- strategy cost
- $354.44
- judge cost
- $9.19
- wall clock
- 9h 48m
- git
- fce5807b2d
Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.