research · web
NIST SP 800-63B password rules, current revision
Pre-registered hypothesis
Hypothesis
A version trap more than a breadth task: every rule sits in one section of one document, but the superseded revision 3, also in the corpus, answers four of them differently. Decomposition has nothing to split, so expect single to match or beat fan-out on score and cost; in a one-run calibration it scored 1.0 with both Haiku 4.5 and Sonnet 5.5.
Written when the task was written, before any run. It is a prediction in words, so read it against what happened rather than trusting a verdict drawn for you.
What happened on Haiku 4.5
- Highest mean score: Single 1.000
- Lowest: Gated-execute 1.000
Means over the runs that count; intervals and verdicts are on the results page.
Who solved it
On Haiku 4.5, every architecture averages 1.000: this task does not separate them. The same task, every architecture, every model: one square per run.
Point at a square or tab to it to read its run. Each square opens its run.
The prompt every architecture was handed
What does the current revision of NIST SP 800-63B require of verifiers and credential service providers about passwords? For each topic below, give the normative keyword of the requirement and the number it sets, if any. - single-factor-minimum-length: minimum length of a password used as the only authentication factor (characters) - maximum-length: the maximum password length verifiers permit (characters) - composition-rules: imposing composition rules, such as requiring a mix of character types - periodic-change: requiring subscribers to change their passwords periodically - password-hints: letting subscribers store a password hint that unauthenticated claimants can see - security-questions: prompting for knowledge-based authentication or security questions when a password is chosen - blocklist: checking a new password against a blocklist of commonly used, expected or compromised passwords - password-managers: allowing password managers and autofill - salt-length: the length of the salt used when storing passwords (bits) - hashing-scheme: using an approved password hashing scheme from the latest revision of SP 800-132 Research this with the provided web tools. Report, for every topic: - keyword: the normative keyword the current requirement uses - value: the number the requirement sets (in characters or bits), or null when it sets none - quote: a sentence from the current revision of SP 800-63B that states the requirement, copied word for word (at least six words). - url: the URL of the page the quote comes from, which you opened with fetch_page. Write whatever explanation you like, then end the answer with one fenced JSON block of exactly this shape, with one entry per topic: ```json {"answers":[{"topic":"single-factor-minimum-length","keyword":"SHALL | SHALL NOT | SHOULD | SHOULD NOT | MAY","value":"…","quote":"<a sentence copied word for word>","url":"<the URL of that page, as you opened it>"}]} ``` Quotes are checked against the text of the page at their URL, so copy them exactly. The block must be valid JSON: write a double quote inside a string as \".
The domain policy
Appended to the system prompt for every architecture, so none of them gets a rules advantage.
Research rules: - The `search` and `fetch_page` tools are your only access to the web. Any other browsing tool is unavailable. - Search snippets are previews, not sources. Open a page with `fetch_page` before relying on what it says. - Long pages arrive one window at a time. Continue with `offset`, or use `find` to jump to a phrase. - Cite the URL of every source you rely on, next to the claim it supports. - Never cite a URL you did not open with `fetch_page`. A citation to an unopened page is scored as a fabrication. - Quotes must be copied word for word from the page you cite; they are checked against the page text. - Quote only passages the tools have shown you in this session. A quote recalled from memory does not count, even when the page contains it. - When sources disagree, say so explicitly and attribute each position, instead of silently picking one.
The traps it sets
A trap is a specific way to go wrong, declared before any run and tied to the assertions or rules that register it. A trap fired in a run when one of those failed. On Haiku 4.5:
Where it was lost
On Haiku 4.5, no architecture fell short of any graded item. A score says an architecture lost points. This says which ones: for each graded item, the runs that earned less than full credit, out of the runs that count.
In numbers, on Claude Haiku 4.5
- Gated-execute1.000
2 runs100% pass$0.070 / run57.2s
- Orchestrator-workers1.000
2 runs100% pass$0.220 / run1m 25s
- Plan-execute1.000
2 runs100% pass$0.105 / run1m 21s
- Plan-execute-verify1.000
2 runs100% pass$0.175 / run2m 7s
- Propose-select1.000
2 runs100% pass$0.218 / run1m 28s
- Single1.000
2 runs100% pass$0.053 / run34.9s
| Strategy | n | Mean score | Pass | Cost / run | Latency | Failure modes |
|---|---|---|---|---|---|---|
| Gated-execute | 2 | 1.000 | 100% | $0.070 | 57.2s | — |
| Orchestrator-workers | 2 | 1.000 | 100% | $0.220 | 1m 25s | — |
| Plan-execute | 2 | 1.000 | 100% | $0.105 | 1m 21s | — |
| Plan-execute-verify | 2 | 1.000 | 100% | $0.175 | 2m 7s | — |
| Propose-select | 2 | 1.000 | 100% | $0.218 | 1m 28s | — |
| Single | 2 | 1.000 | 100% | $0.053 | 34.9s | — |
Every run on Haiku 4.512 runs · $1.68 spent
Provenance of the numbers on this page
- run set
- scaffold-or-scale-v1
- models
- Haiku 4.5, Sonnet 5.5, Opus 5.5
- effort
- high
- repeats
- 2
- fixtures
- replay
- runs
- 900 / 900
- strategy cost
- $354.44
- judge cost
- $9.19
- wall clock
- 9h 48m
- git
- fce5807b2d
Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.