ScaffoldBench

research · web

OWASP vs NIST on password hashing

hardsecuritystandardsversions
Task id
research-password-hashing
Turn limit
38
Runs on Opus 5.5
12
Pass rate
83%

Pre-registered hypothesis

Hypothesis

Plan-and-verify architectures should lead: the difficulty is version and nuance, not breadth. A verifier that rereads the current NIST text catches revision-3 wording; parallel workers add little.

Written when the task was written, before any run. It is a prediction in words, so read it against what happened rather than trusting a verdict drawn for you.

What happened on Opus 5.5

  • Highest mean score: Single 1.000
  • Lowest: Plan-execute-verify 0.906

Means over the runs that count; intervals and verdicts are on the results page.

Who solved it

On Opus 5.5, Single, Plan-execute, Orchestrator-workers, Propose-select and Gated-execute score highest at 1.000 and Plan-execute-verify lowest at 0.906. The same task, every architecture, every model: one square per run.

Haiku 4.5
Sonnet 5.5
Opus 5.5
Single
0.773 mean score of the runs that count
0.875 mean score of the runs that count
1.000 mean score of the runs that count
Plan-execute
0.875 mean score of the runs that count
0.906 mean score of the runs that count
1.000 mean score of the runs that count
Orchestrator-workers
0.875 mean score of the runs that count
0.875 mean score of the runs that count
1.000 mean score of the runs that count
Plan-execute-verify
0.781 mean score of the runs that count
0.875 mean score of the runs that count
0.906 mean score of the runs that count
Propose-select
0.727 mean score of the runs that count
0.875 mean score of the runs that count
1.000 mean score of the runs that count
Gated-execute
0.758 mean score of the runs that count
0.875 mean score of the runs that count
1.000 mean score of the runs that count

Point at a square or tab to it to read its run. Each square opens its run.

passedpartial creditfailedexcluded: integrity check failedexcluded: API errorOne square per run; each opens its run.

The prompt every architecture was handed

A team must choose a password hashing algorithm for a web service that also has to satisfy US federal (FIPS) requirements. What does the OWASP Password Storage guidance recommend, what does the current revision of NIST SP 800-63B require, and how should the team reconcile the two? Report what each source actually says. Research this with the provided web tools and write a complete, self-contained answer. Write the answer for the user in prose. Then list the evidence for your key claims: end the answer with one fenced JSON block of exactly this shape, one entry per claim: ```json {"evidence": [{"claim": "<the claim, in your words>", "quote": "<a sentence copied word for word from the page>", "url": "<the URL of that page, as you opened it>"}]} ``` Every quote is checked against the text of the page at its URL. A paraphrase, or a quote from a page you did not open with fetch_page, does not count as evidence. The block must be valid JSON: write a double quote inside a string as \".

The domain policy

Appended to the system prompt for every architecture, so none of them gets a rules advantage.

Research rules: - The `search` and `fetch_page` tools are your only access to the web. Any other browsing tool is unavailable. - Search snippets are previews, not sources. Open a page with `fetch_page` before relying on what it says. - Long pages arrive one window at a time. Continue with `offset`, or use `find` to jump to a phrase. - Cite the URL of every source you rely on, next to the claim it supports. - Never cite a URL you did not open with `fetch_page`. A citation to an unopened page is scored as a fabrication. - Quotes must be copied word for word from the page you cite; they are checked against the page text. - Quote only passages the tools have shown you in this session. A quote recalled from memory does not count, even when the page contains it. - When sources disagree, say so explicitly and attribute each position, instead of silently picking one.

The traps it sets

A trap is a specific way to go wrong, declared before any run and tied to the assertions or rules that register it. A trap fired in a run when one of those failed. On Opus 5.5:

OWASP vs NIST on password hashing3 traps · runs that fell in, of runs ranked
Trap
Single
Plan-execute
Orchestrator-workers
Plan-execute-verify
Propose-select
Gated-execute
nist-revision-3Reporting revision 3's 'PBKDF2 and Balloon ... a memory-hard function SHOULD be used' as what NIST currently requires. Revision 3 is in the corpus and says it is superseded.
Single fell into nist-revision-3 in 0 of 2 runs
Plan-execute fell into nist-revision-3 in 0 of 2 runs
Orchestrator-workers fell into nist-revision-3 in 0 of 2 runs
Plan-execute-verify fell into nist-revision-3 in 0 of 2 runs
Propose-select fell into nist-revision-3 in 0 of 2 runs
Gated-execute fell into nist-revision-3 in 0 of 2 runs
false-conflictPresenting OWASP and NIST as irreconcilable. OWASP itself prescribes PBKDF2 when FIPS-140 compliance is required.
Single fell into false-conflict in 0 of 2 runs
Plan-execute fell into false-conflict in 0 of 2 runs
Orchestrator-workers fell into false-conflict in 0 of 2 runs
Plan-execute-verify fell into false-conflict in 0 of 2 runs
Propose-select fell into false-conflict in 0 of 2 runs
Gated-execute fell into false-conflict in 0 of 2 runs
strict-rankingReading OWASP's alternatives as a strict ranking rather than conditions (unavailable / legacy / FIPS).
Single fell into strict-ranking in 0 of 2 runs
Plan-execute fell into strict-ranking in 0 of 2 runs
Orchestrator-workers fell into strict-ranking in 0 of 2 runs
Plan-execute-verify fell into strict-ranking in 0 of 2 runs
Propose-select fell into strict-ranking in 0 of 2 runs
Gated-execute fell into strict-ranking in 0 of 2 runs

Where it was lost

On Opus 5.5, the item most often missed is “States that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording.”, by 8% of runs. A score says an architecture lost points. This says which ones: for each graded item, the runs that earned less than full credit, out of the runs that count.

Graded items on Opus 5.5runs that fell short, of runs that count
Graded item
Single
Plan-execute
Orchestrator-workers
Plan-execute-verify
Propose-select
Gated-execute
States that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism.weight 1.5
Single fell short of "States that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism." in 0 of 2 runs
Plan-execute fell short of "States that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism." in 0 of 2 runs
Orchestrator-workers fell short of "States that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism." in 0 of 2 runs
Plan-execute-verify fell short of "States that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism." in 0 of 2 runs
Propose-select fell short of "States that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism." in 0 of 2 runs
Gated-execute fell short of "States that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism." in 0 of 2 runs
States OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required.weight 1
Single fell short of "States OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required." in 0 of 2 runs
Plan-execute fell short of "States OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required." in 0 of 2 runs
Orchestrator-workers fell short of "States OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required." in 0 of 2 runs
Plan-execute-verify fell short of "States OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required." in 0 of 2 runs
Propose-select fell short of "States OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required." in 0 of 2 runs
Gated-execute fell short of "States OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required." in 0 of 2 runs
States that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used.weight 1.5
Single fell short of "States that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used." in 0 of 2 runs
Plan-execute fell short of "States that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used." in 0 of 2 runs
Orchestrator-workers fell short of "States that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used." in 0 of 2 runs
Plan-execute-verify fell short of "States that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used." in 0 of 2 runs
Propose-select fell short of "States that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used." in 0 of 2 runs
Gated-execute fell short of "States that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used." in 0 of 2 runs
States that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording.weight 1
Single fell short of "States that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording." in 0 of 2 runs
Plan-execute fell short of "States that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording." in 0 of 2 runs
Orchestrator-workers fell short of "States that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording." in 0 of 2 runs
Plan-execute-verify fell short of "States that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording." in 1 of 2 runs
Propose-select fell short of "States that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording." in 0 of 2 runs
Gated-execute fell short of "States that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording." in 0 of 2 runs
Explains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply.weight 2
Single fell short of "Explains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply." in 0 of 2 runs
Plan-execute fell short of "Explains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply." in 0 of 2 runs
Orchestrator-workers fell short of "Explains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply." in 0 of 2 runs
Plan-execute-verify fell short of "Explains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply." in 0 of 2 runs
Propose-select fell short of "Explains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply." in 0 of 2 runs
Gated-execute fell short of "Explains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply." in 0 of 2 runs
States at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash.weight 1
Single fell short of "States at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash." in 0 of 2 runs
Plan-execute fell short of "States at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash." in 0 of 2 runs
Orchestrator-workers fell short of "States at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash." in 0 of 2 runs
Plan-execute-verify fell short of "States at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash." in 0 of 2 runs
Propose-select fell short of "States at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash." in 0 of 2 runs
Gated-execute fell short of "States at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash." in 0 of 2 runs
Cited pages that were never openedA rule: breaching it multiplies the score down.
Single fell short of "Cited pages that were never opened" in 0 of 2 runs
Plan-execute fell short of "Cited pages that were never opened" in 0 of 2 runs
Orchestrator-workers fell short of "Cited pages that were never opened" in 0 of 2 runs
Plan-execute-verify fell short of "Cited pages that were never opened" in 1 of 2 runs
Propose-select fell short of "Cited pages that were never opened" in 0 of 2 runs
Gated-execute fell short of "Cited pages that were never opened" in 0 of 2 runs

How the failures failed1hallucinated citation1wrong answer

In numbers, on Claude Opus 5.5

  • Gated-execute
    1.000

    2 runs100% pass$0.264 / run51.4s

  • Orchestrator-workers
    1.000

    2 runs100% pass$1.21 / run2m 17s

  • Plan-execute
    1.000

    2 runs100% pass$0.558 / run1m 46s

  • Propose-select
    1.000

    2 runs100% pass$1.28 / run1m 51s

  • Single
    1.000

    2 runs100% pass$0.262 / run47.2s

  • Plan-execute-verify
    0.906

    2 runs0% pass$0.687 / run1m 59s

    hallucinated citation ×1, wrong answer ×1

Every run on Opus 5.512 runs · $8.53 spent
RunStrategyRepeatScoreResultCostLatency
claude-opus-5-5__gated-execute__research-password-hashing__r1Gated-execute11.000pass$0.24346.4s
claude-opus-5-5__gated-execute__research-password-hashing__r2Gated-execute21.000pass$0.28456.4s
claude-opus-5-5__orchestrator-workers__research-password-hashing__r1Orchestrator-workers11.000pass$1.242m 20s
claude-opus-5-5__orchestrator-workers__research-password-hashing__r2Orchestrator-workers21.000pass$1.192m 15s
claude-opus-5-5__plan-execute-verify__research-password-hashing__r1Plan-execute-verify10.875hallucinated citation$0.7112m 5s
claude-opus-5-5__plan-execute-verify__research-password-hashing__r2Plan-execute-verify20.938wrong answer$0.6631m 53s
claude-opus-5-5__plan-execute__research-password-hashing__r1Plan-execute11.000pass$0.5091m 40s
claude-opus-5-5__plan-execute__research-password-hashing__r2Plan-execute21.000pass$0.6061m 53s
claude-opus-5-5__propose-select__research-password-hashing__r1Propose-select11.000pass$1.271m 50s
claude-opus-5-5__propose-select__research-password-hashing__r2Propose-select21.000pass$1.291m 51s
claude-opus-5-5__single__research-password-hashing__r1Single11.000pass$0.27849.5s
claude-opus-5-5__single__research-password-hashing__r2Single21.000pass$0.24644.8s

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.