ScaffoldBench

research · web

NIST SP 800-63B password rules, current revision

mediumsecuritystandardsbreadthversionsgold-key
Task id
research-nist-password-rules
Turn limit
30
Runs on Haiku 4.5
12
Pass rate
100%

Pre-registered hypothesis

Hypothesis

A version trap more than a breadth task: every rule sits in one section of one document, but the superseded revision 3, also in the corpus, answers four of them differently. Decomposition has nothing to split, so expect single to match or beat fan-out on score and cost; in a one-run calibration it scored 1.0 with both Haiku 4.5 and Sonnet 5.5.

Written when the task was written, before any run. It is a prediction in words, so read it against what happened rather than trusting a verdict drawn for you.

What happened on Haiku 4.5

  • Highest mean score: Single 1.000
  • Lowest: Gated-execute 1.000

Means over the runs that count; intervals and verdicts are on the results page.

Who solved it

On Haiku 4.5, every architecture averages 1.000: this task does not separate them. The same task, every architecture, every model: one square per run.

1.000 mean score of the runs that count
1.000 mean score of the runs that count
0.875 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
0.750 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
0.875 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
0.792 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count
1.000 mean score of the runs that count

Point at a square or tab to it to read its run. Each square opens its run.

passedpartial creditfailedexcluded: integrity check failedexcluded: API errorOne square per run; each opens its run.

The prompt every architecture was handed

What does the current revision of NIST SP 800-63B require of verifiers and credential service providers about passwords? For each topic below, give the normative keyword of the requirement and the number it sets, if any. - single-factor-minimum-length: minimum length of a password used as the only authentication factor (characters) - maximum-length: the maximum password length verifiers permit (characters) - composition-rules: imposing composition rules, such as requiring a mix of character types - periodic-change: requiring subscribers to change their passwords periodically - password-hints: letting subscribers store a password hint that unauthenticated claimants can see - security-questions: prompting for knowledge-based authentication or security questions when a password is chosen - blocklist: checking a new password against a blocklist of commonly used, expected or compromised passwords - password-managers: allowing password managers and autofill - salt-length: the length of the salt used when storing passwords (bits) - hashing-scheme: using an approved password hashing scheme from the latest revision of SP 800-132 Research this with the provided web tools. Report, for every topic: - keyword: the normative keyword the current requirement uses - value: the number the requirement sets (in characters or bits), or null when it sets none - quote: a sentence from the current revision of SP 800-63B that states the requirement, copied word for word (at least six words). - url: the URL of the page the quote comes from, which you opened with fetch_page. Write whatever explanation you like, then end the answer with one fenced JSON block of exactly this shape, with one entry per topic: ```json {"answers":[{"topic":"single-factor-minimum-length","keyword":"SHALL | SHALL NOT | SHOULD | SHOULD NOT | MAY","value":"…","quote":"<a sentence copied word for word>","url":"<the URL of that page, as you opened it>"}]} ``` Quotes are checked against the text of the page at their URL, so copy them exactly. The block must be valid JSON: write a double quote inside a string as \".

The domain policy

Appended to the system prompt for every architecture, so none of them gets a rules advantage.

Research rules: - The `search` and `fetch_page` tools are your only access to the web. Any other browsing tool is unavailable. - Search snippets are previews, not sources. Open a page with `fetch_page` before relying on what it says. - Long pages arrive one window at a time. Continue with `offset`, or use `find` to jump to a phrase. - Cite the URL of every source you rely on, next to the claim it supports. - Never cite a URL you did not open with `fetch_page`. A citation to an unopened page is scored as a fabrication. - Quotes must be copied word for word from the page you cite; they are checked against the page text. - Quote only passages the tools have shown you in this session. A quote recalled from memory does not count, even when the page contains it. - When sources disagree, say so explicitly and attribute each position, instead of silently picking one.

The traps it sets

A trap is a specific way to go wrong, declared before any run and tied to the assertions or rules that register it. A trap fired in a run when one of those failed. On Haiku 4.5:

NIST SP 800-63B password rules, current revision2 traps · runs that fell in, of runs ranked
Trap
Single
Plan-execute
Orchestrator-workers
Plan-execute-verify
Propose-select
Gated-execute
revision-3Revision 3 sets an 8-character minimum, says composition rules and periodic changes SHOULD NOT (not SHALL NOT) be imposed, and only SHOULD permit paste. It is in the corpus, marked superseded.
Single fell into revision-3 in 0 of 2 runs
Plan-execute fell into revision-3 in 0 of 2 runs
Orchestrator-workers fell into revision-3 in 0 of 2 runs
Plan-execute-verify fell into revision-3 in 0 of 2 runs
Propose-select fell into revision-3 in 0 of 2 runs
Gated-execute fell into revision-3 in 0 of 2 runs
owasp-as-nistAnswering from OWASP's Authentication Cheat Sheet, which paraphrases NIST in its own words ("verifiers should not mandate arbitrary password changes") and so cannot settle which keyword NIST uses.
Single fell into owasp-as-nist in 0 of 2 runs
Plan-execute fell into owasp-as-nist in 0 of 2 runs
Orchestrator-workers fell into owasp-as-nist in 0 of 2 runs
Plan-execute-verify fell into owasp-as-nist in 0 of 2 runs
Propose-select fell into owasp-as-nist in 0 of 2 runs
Gated-execute fell into owasp-as-nist in 0 of 2 runs

Where it was lost

On Haiku 4.5, no architecture fell short of any graded item. A score says an architecture lost points. This says which ones: for each graded item, the runs that earned less than full credit, out of the runs that count.

Graded items on Haiku 4.5runs that fell short, of runs that count
Graded item
Single
Plan-execute
Orchestrator-workers
Plan-execute-verify
Propose-select
Gated-execute
topic single-factor-minimum-length: minimum length of a password used as the only authentication factor (characters)weight 1
Single fell short of "topic single-factor-minimum-length: minimum length of a password used as the only authentication factor (characters)" in 0 of 2 runs
Plan-execute fell short of "topic single-factor-minimum-length: minimum length of a password used as the only authentication factor (characters)" in 0 of 2 runs
Orchestrator-workers fell short of "topic single-factor-minimum-length: minimum length of a password used as the only authentication factor (characters)" in 0 of 2 runs
Plan-execute-verify fell short of "topic single-factor-minimum-length: minimum length of a password used as the only authentication factor (characters)" in 0 of 2 runs
Propose-select fell short of "topic single-factor-minimum-length: minimum length of a password used as the only authentication factor (characters)" in 0 of 2 runs
Gated-execute fell short of "topic single-factor-minimum-length: minimum length of a password used as the only authentication factor (characters)" in 0 of 2 runs
topic maximum-length: the maximum password length verifiers permit (characters)weight 1
Single fell short of "topic maximum-length: the maximum password length verifiers permit (characters)" in 0 of 2 runs
Plan-execute fell short of "topic maximum-length: the maximum password length verifiers permit (characters)" in 0 of 2 runs
Orchestrator-workers fell short of "topic maximum-length: the maximum password length verifiers permit (characters)" in 0 of 2 runs
Plan-execute-verify fell short of "topic maximum-length: the maximum password length verifiers permit (characters)" in 0 of 2 runs
Propose-select fell short of "topic maximum-length: the maximum password length verifiers permit (characters)" in 0 of 2 runs
Gated-execute fell short of "topic maximum-length: the maximum password length verifiers permit (characters)" in 0 of 2 runs
topic composition-rules: imposing composition rules, such as requiring a mix of character typesweight 1
Single fell short of "topic composition-rules: imposing composition rules, such as requiring a mix of character types" in 0 of 2 runs
Plan-execute fell short of "topic composition-rules: imposing composition rules, such as requiring a mix of character types" in 0 of 2 runs
Orchestrator-workers fell short of "topic composition-rules: imposing composition rules, such as requiring a mix of character types" in 0 of 2 runs
Plan-execute-verify fell short of "topic composition-rules: imposing composition rules, such as requiring a mix of character types" in 0 of 2 runs
Propose-select fell short of "topic composition-rules: imposing composition rules, such as requiring a mix of character types" in 0 of 2 runs
Gated-execute fell short of "topic composition-rules: imposing composition rules, such as requiring a mix of character types" in 0 of 2 runs
topic periodic-change: requiring subscribers to change their passwords periodicallyweight 1
Single fell short of "topic periodic-change: requiring subscribers to change their passwords periodically" in 0 of 2 runs
Plan-execute fell short of "topic periodic-change: requiring subscribers to change their passwords periodically" in 0 of 2 runs
Orchestrator-workers fell short of "topic periodic-change: requiring subscribers to change their passwords periodically" in 0 of 2 runs
Plan-execute-verify fell short of "topic periodic-change: requiring subscribers to change their passwords periodically" in 0 of 2 runs
Propose-select fell short of "topic periodic-change: requiring subscribers to change their passwords periodically" in 0 of 2 runs
Gated-execute fell short of "topic periodic-change: requiring subscribers to change their passwords periodically" in 0 of 2 runs
topic password-hints: letting subscribers store a password hint that unauthenticated claimants can seeweight 1
Single fell short of "topic password-hints: letting subscribers store a password hint that unauthenticated claimants can see" in 0 of 2 runs
Plan-execute fell short of "topic password-hints: letting subscribers store a password hint that unauthenticated claimants can see" in 0 of 2 runs
Orchestrator-workers fell short of "topic password-hints: letting subscribers store a password hint that unauthenticated claimants can see" in 0 of 2 runs
Plan-execute-verify fell short of "topic password-hints: letting subscribers store a password hint that unauthenticated claimants can see" in 0 of 2 runs
Propose-select fell short of "topic password-hints: letting subscribers store a password hint that unauthenticated claimants can see" in 0 of 2 runs
Gated-execute fell short of "topic password-hints: letting subscribers store a password hint that unauthenticated claimants can see" in 0 of 2 runs
topic security-questions: prompting for knowledge-based authentication or security questions when a password is chosenweight 1
Single fell short of "topic security-questions: prompting for knowledge-based authentication or security questions when a password is chosen" in 0 of 2 runs
Plan-execute fell short of "topic security-questions: prompting for knowledge-based authentication or security questions when a password is chosen" in 0 of 2 runs
Orchestrator-workers fell short of "topic security-questions: prompting for knowledge-based authentication or security questions when a password is chosen" in 0 of 2 runs
Plan-execute-verify fell short of "topic security-questions: prompting for knowledge-based authentication or security questions when a password is chosen" in 0 of 2 runs
Propose-select fell short of "topic security-questions: prompting for knowledge-based authentication or security questions when a password is chosen" in 0 of 2 runs
Gated-execute fell short of "topic security-questions: prompting for knowledge-based authentication or security questions when a password is chosen" in 0 of 2 runs
topic blocklist: checking a new password against a blocklist of commonly used, expected or compromised passwordsweight 1
Single fell short of "topic blocklist: checking a new password against a blocklist of commonly used, expected or compromised passwords" in 0 of 2 runs
Plan-execute fell short of "topic blocklist: checking a new password against a blocklist of commonly used, expected or compromised passwords" in 0 of 2 runs
Orchestrator-workers fell short of "topic blocklist: checking a new password against a blocklist of commonly used, expected or compromised passwords" in 0 of 2 runs
Plan-execute-verify fell short of "topic blocklist: checking a new password against a blocklist of commonly used, expected or compromised passwords" in 0 of 2 runs
Propose-select fell short of "topic blocklist: checking a new password against a blocklist of commonly used, expected or compromised passwords" in 0 of 2 runs
Gated-execute fell short of "topic blocklist: checking a new password against a blocklist of commonly used, expected or compromised passwords" in 0 of 2 runs
topic password-managers: allowing password managers and autofillweight 1
Single fell short of "topic password-managers: allowing password managers and autofill" in 0 of 2 runs
Plan-execute fell short of "topic password-managers: allowing password managers and autofill" in 0 of 2 runs
Orchestrator-workers fell short of "topic password-managers: allowing password managers and autofill" in 0 of 2 runs
Plan-execute-verify fell short of "topic password-managers: allowing password managers and autofill" in 0 of 2 runs
Propose-select fell short of "topic password-managers: allowing password managers and autofill" in 0 of 2 runs
Gated-execute fell short of "topic password-managers: allowing password managers and autofill" in 0 of 2 runs
topic salt-length: the length of the salt used when storing passwords (bits)weight 1
Single fell short of "topic salt-length: the length of the salt used when storing passwords (bits)" in 0 of 2 runs
Plan-execute fell short of "topic salt-length: the length of the salt used when storing passwords (bits)" in 0 of 2 runs
Orchestrator-workers fell short of "topic salt-length: the length of the salt used when storing passwords (bits)" in 0 of 2 runs
Plan-execute-verify fell short of "topic salt-length: the length of the salt used when storing passwords (bits)" in 0 of 2 runs
Propose-select fell short of "topic salt-length: the length of the salt used when storing passwords (bits)" in 0 of 2 runs
Gated-execute fell short of "topic salt-length: the length of the salt used when storing passwords (bits)" in 0 of 2 runs
topic hashing-scheme: using an approved password hashing scheme from the latest revision of SP 800-132weight 1
Single fell short of "topic hashing-scheme: using an approved password hashing scheme from the latest revision of SP 800-132" in 0 of 2 runs
Plan-execute fell short of "topic hashing-scheme: using an approved password hashing scheme from the latest revision of SP 800-132" in 0 of 2 runs
Orchestrator-workers fell short of "topic hashing-scheme: using an approved password hashing scheme from the latest revision of SP 800-132" in 0 of 2 runs
Plan-execute-verify fell short of "topic hashing-scheme: using an approved password hashing scheme from the latest revision of SP 800-132" in 0 of 2 runs
Propose-select fell short of "topic hashing-scheme: using an approved password hashing scheme from the latest revision of SP 800-132" in 0 of 2 runs
Gated-execute fell short of "topic hashing-scheme: using an approved password hashing scheme from the latest revision of SP 800-132" in 0 of 2 runs

In numbers, on Claude Haiku 4.5

  • Gated-execute
    1.000

    2 runs100% pass$0.070 / run57.2s

  • Orchestrator-workers
    1.000

    2 runs100% pass$0.220 / run1m 25s

  • Plan-execute
    1.000

    2 runs100% pass$0.105 / run1m 21s

  • Plan-execute-verify
    1.000

    2 runs100% pass$0.175 / run2m 7s

  • Propose-select
    1.000

    2 runs100% pass$0.218 / run1m 28s

  • Single
    1.000

    2 runs100% pass$0.053 / run34.9s

Every run on Haiku 4.512 runs · $1.68 spent

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.