RunSonnet 5.5repeat 1
Plan-execute-verify on OWASP vs NIST on password hashing
claude-sonnet-5-5__plan-execute-verify__research-password-hashing__r1
Score
0.875
- Cost
- $0.279
- + $0.093 judging
- Latency
- 1m 11s
- setup 0.0s, grading 5.6s
- Turns
- 27
- Tool calls
- 22
- Subagents
- 0
- Tokens in / out
- 154k / 9.0k
- 106,469 read from cache
Did the design run?
- Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
- No API failures: heldharness.apiNo model call failed at the API.
- Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
- No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
- Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
- No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
- Builtin tools: heldharness.builtin-toolsA check recorded with the run.
- Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
- No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
- Judge answered: heldharness.judgeAt least one grading call returned a usable verdict.
- The plan was a plan: heldplan-execute-verify.planThe planner’s reply was read through its schema and held at least one step.
- Every verdict was read: heldplan-execute-verify.verdictsEvery review round ended with a verdict read from the schema reply, not a default.
- The loop did what its verdicts said: heldplan-execute-verify.loopEvery verdict that called for a revision got one, and nothing was revised without such a verdict.
- Each call was shown the right tools: heldplan-execute-verify.toolsThe planner was shown no tools, the verifier the read-only ones, and the executor and reviser the full toolset.
Phases
Drawn from the phase records: timing and cost per call, no tool calls or messages.
| Phase | Duration | Cost | Turns | Tools | Model |
|---|---|---|---|---|---|
| Plan | 12.8s | $0.020 | 2 | 0 | claude-sonnet-5-5 |
| Execute | 39.2s | $0.158 | 13 | 12 | claude-sonnet-5-5 |
| Verify | 18.6s | $0.100 | 12 | 10 | claude-sonnet-5-5 |
The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.
Runs you can replay
- A gate that blocked the right moveGated-execute · Haiku 4.5
- Three proposals beat one agentPropose-select · Haiku 4.5
- A verifier that earned its keepPlan-execute-verify · Sonnet 5.5
- A revision that did not helpPlan-execute-verify · Haiku 4.5
- The control, missing a stepSingle · Haiku 4.5
Record this cell yourself: npm run bench -- run --strategies plan-execute-verify --tasks research-password-hashing --models claude-sonnet-5-5 --repeats 1
Score breakdown
- passStates that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism.1.00weight 0.19
met, backed by verified evidence E1. “Argon2id, with a minimum of 19 MiB of memory, 2 iterations and 1 degree of parallelism.”
- passStates OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required.1.00weight 0.13
met, backed by verified evidence E2, E3. “scrypt if Argon2id isn't available.”
- passStates that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used.1.00weight 0.19
met, backed by verified evidence E6, E7, E9. “Passwords SHALL be salted and hashed using a suitable password hashing scheme.”
- failStates that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording.0.00weight 0.13
Not stated.
- passExplains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply.1.00weight 0.25
met, backed by verified evidence E2, E4. “OWASP itself carves out the FIPS case.”
- passStates at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash.1.00weight 0.13
met, backed by verified evidence E6, E8, E9. “The cost factor SHOULD be as high as practical and increased over time.”
Penalties
No penalties were applied.
Scorer metadata
- corpus
- {"version":2,"snapshot":"2026-10-01","hash":"e12dc326c69a32d91f85f6355dc63e2a7d672b605685a1ffa1582a42bb1c561e","documents":5}
- opened documents
- ["owasp-password-storage","nist-sp800-63b-4"]
- opened by role
- {"primary":["owasp-password-storage","nist-sp800-63b-4"]}
- displayed share
- {"owasp-password-storage":0.9567,"nist-sp800-63b-4":0.0959}
- cited count
- 2
- verified count
- 2
- hallucinated count
- 0
- citation integrity
- 1
- hallucinated urls
- []
- evidence block
- true
- evidence block repaired
- false
- evidence count
- 13
- verified quote count
- 13
- verified quote rate
- 1
- evidence
- [{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP's first recommendation is Argon2id with a minimum configuration of 19 MiB memory, 2 iterations, 1 degree of parallelism.","quote":"Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP recommends PBKDF2 with at least 600,000 iterations and HMAC-SHA-256 when FIPS-140 compliance is required.","quote":"If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP positions scrypt as the fallback when Argon2id is unavailable and bcrypt only for legacy systems.","quote":"If Argon2id is not available, use scrypt with a minimum CPU/memory cost parameter of (2^17), a minimum block size of 8 (1024 bytes), and a parallelization parameter of 1.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP says PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, so it is preferred when those are required.","quote":"Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP gives per-hash PBKDF2 iteration counts (600,000 for SHA-256, 220,000 for SHA-512).","quote":"PBKDF2-HMAC-SHA512: 220,000 iterations","verified":true,"document":"owasp-password-storage"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"SP 800-63B-4 requires salted and hashed password storage using a suitable password hashing scheme.","quote":"Passwords SHALL be salted and hashed using a suitable password hashing scheme.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST does not name an algorithm but defers to approved schemes in SP 800-132 or updated NIST guidelines.","quote":"An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST requires a salt of at least 32 bits, stored along with the hash.","quote":"The salt SHALL be at least 32 bits in length and chosen to minimize salt value collisions among stored hashes","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST says the cost factor should be as high as practical and increased over time.","quote":"The chosen cost factor SHOULD be as high as practical without negatively impacting verifier performance.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST recommends an additional keyed hash/encryption iteration using a secret key stored separately, ideally in an HSM or TEE (analogous to a pepper).","quote":"The secret key value SHALL be stored separately from the hashed passwords.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"Federal-agency verifiers must use FIPS 140 Level 1-validated cryptography even at AAL1.","quote":"Cryptography used by verifiers operated by or on behalf of federal agencies at AAL1 SHALL be validated to meet the requirements of [FIPS140] Level 1.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"FIPS 140 requirements are satisfied by cryptography validated under the CMVP.","quote":"Throughout this document, [FIPS140] requirements are satisfied by security technologies, products, and services that utilize implementations of cryptography validated by the Cryptography Module Validation Program [CMVP].","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"SP 800-63B-4 is the current revision and supersedes the earlier SP 800-63B.","quote":"This publication supersedes NIST Special Publication (SP) 800-63B.","verified":true,"document":"nist-sp800-63b-4"}]
- judge unverified quotes
- 0
- judge verdicts
- [{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2","E3"]},"nist-current":{"status":"met","support":["E6","E7","E9"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E2","E4"]},"common-ground":{"status":"met","support":["E6","E8","E9"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2","E3"]},"nist-current":{"status":"met","support":["E6","E7","E9"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E2","E4"]},"common-ground":{"status":"met","support":["E8","E9"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2","E3"]},"nist-current":{"status":"met","support":["E6","E7","E9"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E2","E4"]},"common-ground":{"status":"met","support":["E6","E8","E9"]}}]
- judge failures
- 0
- grounded weight share
- 0.875 (88%)
Final answer
I've read both primary sources and am now writing the answer. I didn't open SP 800-132, so nothing below is cited to it.
OWASP Password Storage Cheat Sheet (https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html)
- First choice: Argon2id, with a minimum of 19 MiB of memory, 2 iterations and 1 degree of parallelism. The page lists equivalent trade-offs, such as m=47104, t=1, p=1.
Fallbacks:
- scrypt if Argon2id isn't available.
- bcrypt only for legacy systems, with a work factor of at least 10 and a 72-byte password limit.
FIPS case: "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256." Its PBKDF2 section gives these iteration counts:
- HMAC-SHA256: 600,000, marked as recommended.
- HMAC-SHA512: 220,000.
- HMAC-SHA1: 1,400,000, legacy only.
- Pepper: OWASP suggests considering one for defense in depth.
- Why PBKDF2: OWASP says PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations.
NIST SP 800-63B-4 (https://pages.nist.gov/800-63-4/sp800-63b.html)
- Which revision: This is the current revision. The page's abstract says it "supersedes NIST Special Publication (SP) 800-63B", and the page carries a 26 Aug 2025 date.
What it requires for passwords:
- Verifiers SHALL store passwords in a form resistant to offline attacks.
- Passwords SHALL be salted and hashed using a suitable password hashing scheme.
- The salt SHALL be at least 32 bits and chosen to minimize collisions.
- The salt and the hash SHALL both be stored.
- The cost factor SHOULD be as high as practical and increased over time.
- A reference to the scheme and cost factor SHOULD be stored, to allow migration.
- Approved schemes: It does not name an algorithm. It says "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used." The page contains no occurrence of "PBKDF", and its only hit for "Argon" is part of an unrelated word. So SP 800-63B-4 neither names PBKDF2 nor mentions Argon2. It points to SP 800-132, which I did not open.
- Keyed hash: Verifiers SHOULD add an extra keyed hash or encryption step using a secret key. That key SHALL come from an approved random bit generator and be stored separately from the hashes, ideally in an HSM or TEE. This is NIST's equivalent of OWASP's pepper.
- FIPS 140: "Cryptography used by verifiers operated by or on behalf of federal agencies at AAL1 SHALL be validated to meet the requirements of [FIPS140] Level 1." The AAL2 requirement is the same, "unless otherwise specified". AAL3 requires Level 1 or higher. The document says FIPS 140 requirements are met by cryptography validated under the CMVP.
Reconciling them
The two sources don't conflict. They answer different questions. OWASP's general preference is Argon2id. NIST sets requirements (salted, cost-based, approved scheme, FIPS-validated cryptography for federal verifiers) without naming Argon2id. OWASP itself carves out the FIPS case.
My recommendation, which is inference and not source text:
- Use PBKDF2-HMAC-SHA256 or SHA-512 if the service falls under the federal FIPS requirement. Set at least OWASP's iteration counts (600,000 for SHA-256, 220,000 for SHA-512). Use a unique random salt of at least 32 bits, and larger in practice. Store the scheme and cost with each hash so you can migrate later.
- Run it in a CMVP-validated cryptographic module. NIST's FIPS language is about validated cryptography for federal verifiers. Check that your library or HSM is validated, because "PBKDF2" alone doesn't satisfy that.
- Add a secret-keyed step. NIST says SHOULD and OWASP says "consider". An HMAC with a key held in an HSM or secrets vault serves both.
- Use Argon2id only where FIPS validation doesn't apply. Examples are components outside the FIPS boundary. Neither source says Argon2id is FIPS-approved, and NIST doesn't list it in this document. Confirm with your compliance assessor, and check the latest SP 800-132 or any newer NIST password-hashing guidance, since NIST defers to those.
One caveat: OWASP's iteration counts come from its own benchmarking, and NIST doesn't prescribe a number. Tune the count for your hardware, and raise it over time as NIST advises.
{"evidence": [
{"claim": "OWASP's first recommendation is Argon2id with a minimum configuration of 19 MiB memory, 2 iterations, 1 degree of parallelism.", "quote": "Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP recommends PBKDF2 with at least 600,000 iterations and HMAC-SHA-256 when FIPS-140 compliance is required.", "quote": "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP positions scrypt as the fallback when Argon2id is unavailable and bcrypt only for legacy systems.", "quote": "If Argon2id is not available, use scrypt with a minimum CPU/memory cost parameter of (2^17), a minimum block size of 8 (1024 bytes), and a parallelization parameter of 1.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP says PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, so it is preferred when those are required.", "quote": "Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP gives per-hash PBKDF2 iteration counts (600,000 for SHA-256, 220,000 for SHA-512).", "quote": "PBKDF2-HMAC-SHA512: 220,000 iterations", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "SP 800-63B-4 requires salted and hashed password storage using a suitable password hashing scheme.", "quote": "Passwords SHALL be salted and hashed using a suitable password hashing scheme.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST does not name an algorithm but defers to approved schemes in SP 800-132 or updated NIST guidelines.", "quote": "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST requires a salt of at least 32 bits, stored along with the hash.", "quote": "The salt SHALL be at least 32 bits in length and chosen to minimize salt value collisions among stored hashes", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST says the cost factor should be as high as practical and increased over time.", "quote": "The chosen cost factor SHOULD be as high as practical without negatively impacting verifier performance.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST recommends an additional keyed hash/encryption iteration using a secret key stored separately, ideally in an HSM or TEE (analogous to a pepper).", "quote": "The secret key value SHALL be stored separately from the hashed passwords.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "Federal-agency verifiers must use FIPS 140 Level 1-validated cryptography even at AAL1.", "quote": "Cryptography used by verifiers operated by or on behalf of federal agencies at AAL1 SHALL be validated to meet the requirements of [FIPS140] Level 1.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "FIPS 140 requirements are satisfied by cryptography validated under the CMVP.", "quote": "Throughout this document, [FIPS140] requirements are satisfied by security technologies, products, and services that utilize implementations of cryptography validated by the Cryptography Module Validation Program [CMVP].", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "SP 800-63B-4 is the current revision and supersedes the earlier SP 800-63B.", "quote": "This publication supersedes NIST Special Publication (SP) 800-63B.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"}
]}Provenance of the numbers on this page
- run set
- scaffold-or-scale-v1
- models
- Haiku 4.5, Sonnet 5.5, Opus 5.5
- effort
- high
- repeats
- 2
- fixtures
- replay
- runs
- 900 / 900
- strategy cost
- $354.44
- judge cost
- $9.19
- wall clock
- 9h 48m
- git
- fce5807b2d
Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.