research · web
OWASP vs NIST on password hashing
hardsecuritystandardsversions
Task id
research-password-hashing
Turn limit
38
Runs on Sonnet 5.5
12
Pass rate
0%
Pre-registered hypothesis
Hypothesis
Plan-and-verify architectures should lead: the difficulty is version and nuance, not breadth. A verifier that rereads the current NIST text catches revision-3 wording; parallel workers add little.
Written when the task was written, before any run. It is a prediction in words, so read it against what happened rather than trusting a verdict drawn for you.
What happened on Sonnet 5.5
- Highest mean score: Plan-execute 0.906
- A single agent: 0.875
- Lowest: Gated-execute 0.875
Means over the runs that count; intervals and verdicts are on the results page.
Who solved it
On Sonnet 5.5, Plan-execute scores highest at 0.906 and Gated-execute lowest at 0.875. The same task, every architecture, every model: one square per run.
Architecture
Point at a square or tab to it to read its run. Each square opens its run.
passedpartial creditfailedexcluded: integrity check failedexcluded: API errorOne square per run; each opens its run.
The prompt every architecture was handed
A team must choose a password hashing algorithm for a web service that also has to satisfy US federal (FIPS) requirements. What does the OWASP Password Storage guidance recommend, what does the current revision of NIST SP 800-63B require, and how should the team reconcile the two? Report what each source actually says. Research this with the provided web tools and write a complete, self-contained answer. Write the answer for the user in prose. Then list the evidence for your key claims: end the answer with one fenced JSON block of exactly this shape, one entry per claim: ```json {"evidence": [{"claim": "<the claim, in your words>", "quote": "<a sentence copied word for word from the page>", "url": "<the URL of that page, as you opened it>"}]} ``` Every quote is checked against the text of the page at its URL. A paraphrase, or a quote from a page you did not open with fetch_page, does not count as evidence. The block must be valid JSON: write a double quote inside a string as \".
The domain policy
Appended to the system prompt for every architecture, so none of them gets a rules advantage.
Research rules: - The `search` and `fetch_page` tools are your only access to the web. Any other browsing tool is unavailable. - Search snippets are previews, not sources. Open a page with `fetch_page` before relying on what it says. - Long pages arrive one window at a time. Continue with `offset`, or use `find` to jump to a phrase. - Cite the URL of every source you rely on, next to the claim it supports. - Never cite a URL you did not open with `fetch_page`. A citation to an unopened page is scored as a fabrication. - Quotes must be copied word for word from the page you cite; they are checked against the page text. - Quote only passages the tools have shown you in this session. A quote recalled from memory does not count, even when the page contains it. - When sources disagree, say so explicitly and attribute each position, instead of silently picking one.
The traps it sets
A trap is a specific way to go wrong, declared before any run and tied to the assertions or rules that register it. A trap fired in a run when one of those failed. On Sonnet 5.5:
Trap
Single
Plan-execute
Orchestrator-workers
Plan-execute-verify
Propose-select
Gated-execute
nist-revision-3Reporting revision 3's 'PBKDF2 and Balloon ... a memory-hard function SHOULD be used' as what NIST currently requires. Revision 3 is in the corpus and says it is superseded.
Single fell into nist-revision-3 in 0 of 2 runs
Plan-execute fell into nist-revision-3 in 0 of 2 runs
Orchestrator-workers fell into nist-revision-3 in 0 of 2 runs
Plan-execute-verify fell into nist-revision-3 in 0 of 2 runs
Propose-select fell into nist-revision-3 in 0 of 2 runs
Gated-execute fell into nist-revision-3 in 0 of 2 runs
false-conflictPresenting OWASP and NIST as irreconcilable. OWASP itself prescribes PBKDF2 when FIPS-140 compliance is required.
Single fell into false-conflict in 0 of 2 runs
Plan-execute fell into false-conflict in 0 of 2 runs
Orchestrator-workers fell into false-conflict in 0 of 2 runs
Plan-execute-verify fell into false-conflict in 0 of 2 runs
Propose-select fell into false-conflict in 0 of 2 runs
Gated-execute fell into false-conflict in 0 of 2 runs
strict-rankingReading OWASP's alternatives as a strict ranking rather than conditions (unavailable / legacy / FIPS).
Single fell into strict-ranking in 0 of 2 runs
Plan-execute fell into strict-ranking in 0 of 2 runs
Orchestrator-workers fell into strict-ranking in 0 of 2 runs
Plan-execute-verify fell into strict-ranking in 0 of 2 runs
Propose-select fell into strict-ranking in 0 of 2 runs
Gated-execute fell into strict-ranking in 0 of 2 runs
Where it was lost
On Sonnet 5.5, the item most often missed is “States that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording.”, by 83% of runs. A score says an architecture lost points. This says which ones: for each graded item, the runs that earned less than full credit, out of the runs that count.
Graded item
Single
Plan-execute
Orchestrator-workers
Plan-execute-verify
Propose-select
Gated-execute
States that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism.weight 1.5
Single fell short of "States that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism." in 0 of 2 runs
Plan-execute fell short of "States that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism." in 0 of 2 runs
Orchestrator-workers fell short of "States that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism." in 0 of 2 runs
Plan-execute-verify fell short of "States that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism." in 0 of 2 runs
Propose-select fell short of "States that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism." in 0 of 2 runs
Gated-execute fell short of "States that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism." in 0 of 2 runs
States OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required.weight 1
Single fell short of "States OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required." in 0 of 2 runs
Plan-execute fell short of "States OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required." in 0 of 2 runs
Orchestrator-workers fell short of "States OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required." in 0 of 2 runs
Plan-execute-verify fell short of "States OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required." in 0 of 2 runs
Propose-select fell short of "States OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required." in 0 of 2 runs
Gated-execute fell short of "States OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required." in 0 of 2 runs
States that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used.weight 1.5
Single fell short of "States that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used." in 0 of 2 runs
Plan-execute fell short of "States that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used." in 0 of 2 runs
Orchestrator-workers fell short of "States that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used." in 0 of 2 runs
Plan-execute-verify fell short of "States that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used." in 0 of 2 runs
Propose-select fell short of "States that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used." in 0 of 2 runs
Gated-execute fell short of "States that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used." in 0 of 2 runs
States that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording.weight 1
Single fell short of "States that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording." in 2 of 2 runs
Plan-execute fell short of "States that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording." in 2 of 2 runs
Orchestrator-workers fell short of "States that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording." in 0 of 2 runs
Plan-execute-verify fell short of "States that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording." in 2 of 2 runs
Propose-select fell short of "States that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording." in 2 of 2 runs
Gated-execute fell short of "States that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording." in 2 of 2 runs
Explains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply.weight 2
Single fell short of "Explains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply." in 0 of 2 runs
Plan-execute fell short of "Explains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply." in 0 of 2 runs
Orchestrator-workers fell short of "Explains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply." in 1 of 2 runs
Plan-execute-verify fell short of "Explains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply." in 0 of 2 runs
Propose-select fell short of "Explains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply." in 0 of 2 runs
Gated-execute fell short of "Explains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply." in 0 of 2 runs
States at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash.weight 1
Single fell short of "States at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash." in 0 of 2 runs
Plan-execute fell short of "States at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash." in 0 of 2 runs
Orchestrator-workers fell short of "States at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash." in 0 of 2 runs
Plan-execute-verify fell short of "States at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash." in 0 of 2 runs
Propose-select fell short of "States at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash." in 0 of 2 runs
Gated-execute fell short of "States at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash." in 0 of 2 runs
Cited pages that were never openedA rule: breaching it multiplies the score down.
Single fell short of "Cited pages that were never opened" in 0 of 2 runs
Plan-execute fell short of "Cited pages that were never opened" in 0 of 2 runs
Orchestrator-workers fell short of "Cited pages that were never opened" in 1 of 2 runs
Plan-execute-verify fell short of "Cited pages that were never opened" in 0 of 2 runs
Propose-select fell short of "Cited pages that were never opened" in 0 of 2 runs
Gated-execute fell short of "Cited pages that were never opened" in 0 of 2 runs
How the failures failed11wrong answer1hallucinated citation
In numbers, on Claude Sonnet 5.5
- Plan-execute0.906
2 runs0% pass$0.187 / run48.7s
wrong answer ×2
- Gated-execute0.875
2 runs0% pass$0.101 / run27.1s
wrong answer ×2
- Orchestrator-workers0.875
2 runs0% pass$0.574 / run1m 22s
wrong answer ×1, hallucinated citation ×1
- Plan-execute-verify0.875
2 runs0% pass$0.264 / run1m 3s
wrong answer ×2
- Propose-select0.875
2 runs0% pass$0.454 / run1m 6s
wrong answer ×2
- Single0.875
2 runs0% pass$0.098 / run30.7s
wrong answer ×2
| Strategy | n | Mean score | Pass | Cost / run | Latency | Failure modes |
|---|---|---|---|---|---|---|
| Plan-execute | 2 | 0.906 | 0% | $0.187 | 48.7s | wrong answer ×2 |
| Gated-execute | 2 | 0.875 | 0% | $0.101 | 27.1s | wrong answer ×2 |
| Orchestrator-workers | 2 | 0.875 | 0% | $0.574 | 1m 22s | wrong answer ×1, hallucinated citation ×1 |
| Plan-execute-verify | 2 | 0.875 | 0% | $0.264 | 1m 3s | wrong answer ×2 |
| Propose-select | 2 | 0.875 | 0% | $0.454 | 1m 6s | wrong answer ×2 |
| Single | 2 | 0.875 | 0% | $0.098 | 30.7s | wrong answer ×2 |
Every run on Sonnet 5.512 runs · $3.36 spent
Provenance of the numbers on this page
- run set
- scaffold-or-scale-v1
- models
- Haiku 4.5, Sonnet 5.5, Opus 5.5
- effort
- high
- repeats
- 2
- fixtures
- replay
- runs
- 900 / 900
- strategy cost
- $354.44
- judge cost
- $9.19
- wall clock
- 9h 48m
- git
- fce5807b2d
Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.