RunSonnet 5.5repeat 1
Plan-execute on OWASP vs NIST on password hashing
claude-sonnet-5-5__plan-execute__research-password-hashing__r1
Score
0.875
- Cost
- $0.210
- + $0.106 judging
- Latency
- 51.3s
- setup 0.0s, grading 5.7s
- Turns
- 16
- Tool calls
- 13
- Subagents
- 0
- Tokens in / out
- 124k / 7.3k
- 91,737 read from cache
Did the design run?
- Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
- No API failures: heldharness.apiNo model call failed at the API.
- Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
- No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
- Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
- No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
- Builtin tools: heldharness.builtin-toolsA check recorded with the run.
- Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
- No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
- Judge answered: heldharness.judgeAt least one grading call returned a usable verdict.
- The plan was a plan: heldplan-execute.planThe planner’s reply was read through its schema and held at least one step.
- Each call was shown the right tools: heldplan-execute.toolsThe planner was shown no tools and the executor the task’s full toolset.
Phases
Drawn from the phase records: timing and cost per call, no tool calls or messages.
| Phase | Duration | Cost | Turns | Tools | Model |
|---|---|---|---|---|---|
| Plan | 12.1s | $0.018 | 2 | 0 | claude-sonnet-5-5 |
| Execute | 39.1s | $0.191 | 14 | 13 | claude-sonnet-5-5 |
The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.
Runs you can replay
- A gate that blocked the right moveGated-execute · Haiku 4.5
- Three proposals beat one agentPropose-select · Haiku 4.5
- A verifier that earned its keepPlan-execute-verify · Sonnet 5.5
- A revision that did not helpPlan-execute-verify · Haiku 4.5
- The control, missing a stepSingle · Haiku 4.5
Record this cell yourself: npm run bench -- run --strategies plan-execute --tasks research-password-hashing --models claude-sonnet-5-5 --repeats 1
Score breakdown
- passStates that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism.1.00weight 0.19
met, backed by verified evidence E1. “Its summary says to use Argon2id with a minimum of 19 MiB of memory, 2 iterations and 1 degree of parallelism.”
- passStates OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required.1.00weight 0.13
met, backed by verified evidence E2, E4. “scrypt if Argon2id is unavailable, with N=2^17, r=8, p=1.”
- passStates that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used.1.00weight 0.19
met, backed by verified evidence E7, E8, E9. “An approved scheme from the latest SP 800-132 or updated NIST guidance SHOULD be used.”
- failStates that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording.0.00weight 0.13
Not stated.
- passExplains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply.1.00weight 0.25
met, backed by verified evidence E2, E3. “That is OWASP's own FIPS fallback, and it appears to fit NIST SP 800-63B-4 as well”
- passStates at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash.1.00weight 0.13
met, backed by verified evidence E8, E10, E11, E6. “Store the algorithm and cost with each hash, as both sources recommend”
Penalties
No penalties were applied.
Scorer metadata
- corpus
- {"version":2,"snapshot":"2026-10-01","hash":"e12dc326c69a32d91f85f6355dc63e2a7d672b605685a1ffa1582a42bb1c561e","documents":5}
- opened documents
- ["owasp-password-storage","nist-sp800-63b-4"]
- opened by role
- {"primary":["owasp-password-storage","nist-sp800-63b-4"]}
- displayed share
- {"owasp-password-storage":0.9567,"nist-sp800-63b-4":0.115}
- cited count
- 2
- verified count
- 2
- hallucinated count
- 0
- citation integrity
- 1
- hallucinated urls
- []
- evidence block
- true
- evidence block repaired
- false
- evidence count
- 16
- verified quote count
- 16
- verified quote rate
- 1
- evidence
- [{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP's first-choice recommendation is Argon2id with a minimum configuration of 19 MiB, 2 iterations, parallelism 1.","quote":"Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP says to use PBKDF2 with at least 600,000 iterations and HMAC-SHA-256 when FIPS-140 compliance is required.","quote":"If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP says PBKDF2 is permitted by NIST SP 800-63B-4, has FIPS-140 validated implementations, and should be preferred when those are required.","quote":"Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP lists scrypt as the fallback if Argon2id is unavailable and bcrypt only for legacy systems.","quote":"The bcrypt password hashing function should only be used for password storage in legacy systems where Argon2 and scrypt are not available.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP gives PBKDF2 iteration counts per hash and marks SHA-1 as legacy only.","quote":"PBKDF2-HMAC-SHA256: 600,000 iterations (recommended)","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP advises storing algorithm and work factor with the hash and allowing mixed hashes during migration.","quote":"During the transition period, allow for a mix of old and new hashing algorithms.","verified":true,"document":"owasp-password-storage"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST SP 800-63B-4 supersedes the earlier SP 800-63B, so Revision 4 is current; the page is dated 26 Aug 2025.","quote":"This publication supersedes NIST Special Publication (SP) 800-63B.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST requires passwords to be salted and hashed with a suitable password hashing scheme.","quote":"Passwords SHALL be salted and hashed using a suitable password hashing scheme.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST says an approved scheme from SP 800-132 or updated NIST guidance SHOULD be used (no specific algorithm named), and the cost factor should be as high as practical.","quote":"An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST requires a salt of at least 32 bits and recommends storing the scheme and cost factor to allow migration.","quote":"The salt SHALL be at least 32 bits in length and chosen to minimize salt value collisions among stored hashes","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST recommends storing the scheme and cost factor with each password to allow migration.","quote":"A reference to the password hashing scheme used, including the cost factor, SHOULD be stored for each password to allow migration to new algorithms and work factors.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST recommends an additional keyed hashing step with a secret key stored separately, ideally in hardware-protected storage.","quote":"The secret key value SHALL be stored separately from the hashed passwords.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST sets a 15-character minimum for single-factor passwords and 8 for passwords used only in multi-factor authentication.","quote":"Verifiers and CSPs SHALL require passwords that are used as a single-factor authentication mechanism to be a minimum of 15 characters in length.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST prohibits other composition rules.","quote":"Verifiers and CSPs SHALL NOT impose other composition rules (e.g., requiring mixtures of different character types) for passwords.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"For federal-agency verifiers at AAL1, cryptography must be FIPS 140 Level 1 validated.","quote":"Cryptography used by verifiers operated by or on behalf of federal agencies at AAL1 SHALL be validated to meet the requirements of [FIPS140] Level 1.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST says FIPS 140 requirements are satisfied by implementations validated under the CMVP.","quote":"Throughout this document, [FIPS140] requirements are satisfied by security technologies, products, and services that utilize implementations of cryptography validated by the Cryptography Module Validation Program [CMVP].","verified":true,"document":"nist-sp800-63b-4"}]
- judge unverified quotes
- 0
- judge verdicts
- [{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2","E4"]},"nist-current":{"status":"met","support":["E7","E8","E9"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E8","E10","E11","E6"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2","E4"]},"nist-current":{"status":"met","support":["E7","E8","E9"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E10","E11","E6"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2","E4"]},"nist-current":{"status":"met","support":["E7","E8","E9"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E6","E8","E9","E11"]}}]
- judge failures
- 0
- grounded weight share
- 0.875 (88%)
Final answer
Short answer. If the service falls under FIPS or federal requirements, use PBKDF2 with HMAC-SHA-256 at 600,000 or more iterations, running in a FIPS 140-validated cryptographic module. That is OWASP's own FIPS fallback, and it appears to fit NIST SP 800-63B-4 as well, though NIST's text does not name PBKDF2 itself (see the NIST section). If FIPS does not apply, OWASP's first choice is Argon2id.
What OWASP recommends
Source: https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html. I read the whole page.
- First choice. Its summary says to use Argon2id with a minimum of 19 MiB of memory, 2 iterations and 1 degree of parallelism. The detailed section lists equivalent settings, for example m=47104 (46 MiB) with t=1, or m=12288 (12 MiB) with t=3.
Fallbacks.
- scrypt if Argon2id is unavailable, with N=2^17, r=8, p=1.
- bcrypt only for legacy systems, with a work factor of at least 10 and a 72-byte password limit.
- FIPS case. "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256." The PBKDF2 section adds that PBKDF2 "should be the preferred algorithm when these are required". It lists 600,000 iterations for HMAC-SHA256, 220,000 for HMAC-SHA512, and 1,400,000 for HMAC-SHA1. SHA-1 is marked legacy only, not for new systems.
- Migration. It advises storing the algorithm and work factor with each hash, allowing a mix of old and new hashes during a transition, and re-hashing when users next log in.
- Pepper. It suggests a pepper for defense in depth.
What NIST SP 800-63B-4 requires
Source: https://pages.nist.gov/800-63-4/sp800-63b.html. The page is SP 800-63B-4 and carries a date stamp of 26 Aug 2025. Its abstract says it "supersedes NIST Special Publication (SP) 800-63B", so Revision 4 is the current revision. I did not open the NIST publication PDF, so I did not check any final-publication date beyond this page's stamp.
Hashing and salting (Sec. 3.1.1.2).
- Passwords SHALL be salted and hashed using a suitable password hashing scheme.
- The cost factor SHOULD be as high as practical and increased over time.
- An approved scheme from the latest SP 800-132 or updated NIST guidance SHOULD be used.
- The salt SHALL be at least 32 bits.
- The scheme and cost factor SHOULD be stored with each password to allow migration.
- Verifiers SHOULD also add a keyed-hash or encryption step using a secret key kept separately, ideally in an HSM or TEE.
Length and composition.
- Single-factor passwords must be at least 15 characters. Passwords used only within multi-factor authentication must be at least 8.
- Verifiers SHOULD permit a maximum length of at least 64 characters.
- Other composition rules SHALL NOT be imposed.
- Periodic forced changes SHALL NOT be required.
- Passwords must be checked against a blocklist, and verifiers must rate-limit failed attempts.
- The whole password must be verified, "not truncate it".
- FIPS 140. For federal agencies' verifiers, "Cryptography used by verifiers operated by or on behalf of federal agencies at AAL1 SHALL be validated to meet the requirements of [FIPS140] Level 1." The same applies at AAL2 "unless otherwise specified". At AAL3, verifier cryptography must be validated at Level 1 or higher.
- What NIST does not say. I searched the page and found no mention of PBKDF2, and no mention of Argon2 (the only "Argon" hit is the word "jargon"). It defers to "an approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines". I did not open SP 800-132, so I can't confirm from a source I read what it lists as approved. OWASP says PBKDF2 "is permitted by NIST SP 800-63B-4".
Reconciling the two
- The sources are not in conflict, but they differ in strength and in what they name. NIST's rule that the hashing scheme be "approved" is a SHOULD, and it names no algorithm. OWASP is non-normative guidance that names algorithms. The only place a NIST requirement is expressed as a SHALL for FIPS is the validation of verifier cryptography for federal agencies.
- Choose PBKDF2-HMAC-SHA-256 for the FIPS path. It is the only algorithm in OWASP's list that OWASP ties to FIPS-140 validated implementations and to NIST permission. Use at least 600,000 iterations, and tune upward by benchmarking, since NIST says the cost should be "as high as practical". The password-handling rules above apply regardless of algorithm.
- Use a validated module. NIST's FIPS requirement is about validated cryptography, so use a library or module with a CMVP validation. Whether a given module covers PBKDF2 must be checked against its validation certificate. The pages I read don't say.
- Argon2id is the choice only if FIPS does not apply. That is my inference. Neither page says Argon2id is FIPS-approved, and NIST does not name it. If the service is a federal verifier, or serves agencies, treat Argon2id as not demonstrated to be allowed unless NIST guidance or your assessor says otherwise.
- Plan for migration. Store the algorithm and cost with each hash, as both sources recommend, so you can raise the cost or change algorithm later. NIST says it expects updated guidance on password hashing schemes.
- Optional hardening. Add a secret-keyed HMAC step (a pepper) held in an HSM. Both sources recommend it, and NIST says to store the key separately and ideally in hardware.
Gaps. I did not confirm whether your service is formally in scope for FIPS, which depends on your contract or agency. I did not open SP 800-132, FIPS 140, or any CMVP certificate.
{"evidence": [
{"claim": "OWASP's first-choice recommendation is Argon2id with a minimum configuration of 19 MiB, 2 iterations, parallelism 1.", "quote": "Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP says to use PBKDF2 with at least 600,000 iterations and HMAC-SHA-256 when FIPS-140 compliance is required.", "quote": "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP says PBKDF2 is permitted by NIST SP 800-63B-4, has FIPS-140 validated implementations, and should be preferred when those are required.", "quote": "Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP lists scrypt as the fallback if Argon2id is unavailable and bcrypt only for legacy systems.", "quote": "The bcrypt password hashing function should only be used for password storage in legacy systems where Argon2 and scrypt are not available.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP gives PBKDF2 iteration counts per hash and marks SHA-1 as legacy only.", "quote": "PBKDF2-HMAC-SHA256: 600,000 iterations (recommended)", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP advises storing algorithm and work factor with the hash and allowing mixed hashes during migration.", "quote": "During the transition period, allow for a mix of old and new hashing algorithms.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "NIST SP 800-63B-4 supersedes the earlier SP 800-63B, so Revision 4 is current; the page is dated 26 Aug 2025.", "quote": "This publication supersedes NIST Special Publication (SP) 800-63B.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST requires passwords to be salted and hashed with a suitable password hashing scheme.", "quote": "Passwords SHALL be salted and hashed using a suitable password hashing scheme.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST says an approved scheme from SP 800-132 or updated NIST guidance SHOULD be used (no specific algorithm named), and the cost factor should be as high as practical.", "quote": "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST requires a salt of at least 32 bits and recommends storing the scheme and cost factor to allow migration.", "quote": "The salt SHALL be at least 32 bits in length and chosen to minimize salt value collisions among stored hashes", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST recommends storing the scheme and cost factor with each password to allow migration.", "quote": "A reference to the password hashing scheme used, including the cost factor, SHOULD be stored for each password to allow migration to new algorithms and work factors.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST recommends an additional keyed hashing step with a secret key stored separately, ideally in hardware-protected storage.", "quote": "The secret key value SHALL be stored separately from the hashed passwords.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST sets a 15-character minimum for single-factor passwords and 8 for passwords used only in multi-factor authentication.", "quote": "Verifiers and CSPs SHALL require passwords that are used as a single-factor authentication mechanism to be a minimum of 15 characters in length.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST prohibits other composition rules.", "quote": "Verifiers and CSPs SHALL NOT impose other composition rules (e.g., requiring mixtures of different character types) for passwords.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "For federal-agency verifiers at AAL1, cryptography must be FIPS 140 Level 1 validated.", "quote": "Cryptography used by verifiers operated by or on behalf of federal agencies at AAL1 SHALL be validated to meet the requirements of [FIPS140] Level 1.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST says FIPS 140 requirements are satisfied by implementations validated under the CMVP.", "quote": "Throughout this document, [FIPS140] requirements are satisfied by security technologies, products, and services that utilize implementations of cryptography validated by the Cryptography Module Validation Program [CMVP].", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"}
]}Provenance of the numbers on this page
- run set
- scaffold-or-scale-v1
- models
- Haiku 4.5, Sonnet 5.5, Opus 5.5
- effort
- high
- repeats
- 2
- fixtures
- replay
- runs
- 900 / 900
- strategy cost
- $354.44
- judge cost
- $9.19
- wall clock
- 9h 48m
- git
- fce5807b2d
Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.