RunSonnet 5.5repeat 1
Propose-select on OWASP vs NIST on password hashing
claude-sonnet-5-5__propose-select__research-password-hashing__r1
Score
0.875
- Cost
- $0.484
- + $0.094 judging
- Latency
- 1m 17s
- setup 0.0s, grading 5.5s
- Turns
- 34
- Tool calls
- 29
- Subagents
- 3
- Tokens in / out
- 249k / 19k
- 173,777 read from cache
Did the design run?
- Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
- No API failures: heldharness.apiNo model call failed at the API.
- Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
- No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
- Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
- No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
- Builtin tools: heldharness.builtin-toolsA check recorded with the run.
- Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
- No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
- Judge answered: heldharness.judgeAt least one grading call returned a usable verdict.
- All three proposers ran: heldpropose-select.proposalsThree proposers ran in parallel, each from its own angle, and none was starved while another ran.
- The selector chose: heldpropose-select.selectionThe selector picked a proposal by a valid number, read from its schema reply and not a default.
- Each call was shown the right tools: heldpropose-select.toolsEach proposer was shown the read-only tools, the selector none, and the executor the full toolset.
Phases
Drawn from the phase records: timing and cost per call, no tool calls or messages.
| Phase | Duration | Cost | Turns | Tools | Model |
|---|---|---|---|---|---|
| Proposal 1 (policy-first) | 43.0s | $0.130 | 11 | 10 | claude-sonnet-5-5 |
| Proposal 2 (state-first) | 31.7s | $0.134 | 11 | 10 | claude-sonnet-5-5 |
| Proposal 3 (outcome-first) | 31.8s | $0.120 | 10 | 9 | claude-sonnet-5-5 |
| Select | 33.9s | $0.100 | 2 | 0 | claude-sonnet-5-5 |
The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.
Runs you can replay
- A gate that blocked the right moveGated-execute · Haiku 4.5
- Three proposals beat one agentPropose-select · Haiku 4.5
- A verifier that earned its keepPlan-execute-verify · Sonnet 5.5
- A revision that did not helpPlan-execute-verify · Haiku 4.5
- The control, missing a stepSingle · Haiku 4.5
Record this cell yourself: npm run bench -- run --strategies propose-select --tasks research-password-hashing --models claude-sonnet-5-5 --repeats 1
Score breakdown
- passStates that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism.1.00weight 0.19
met, backed by verified evidence E1. “The default is Argon2id with a minimum of 19 MiB of memory, 2 iterations and 1 degree of parallelism.”
- passStates OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required.1.00weight 0.13
met, backed by verified evidence E2, E4. “If Argon2id is unavailable, use scrypt. Use bcrypt only for legacy systems.”
- passStates that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used.1.00weight 0.19
met, backed by verified evidence E5, E6. “An approved scheme from the latest SP 800-132, or from updated NIST password-hashing guidance, SHOULD be used.”
- failStates that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording.0.00weight 0.13
Not stated.
- passExplains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply.1.00weight 0.25
met, backed by verified evidence E2, E3. “The two sources do not truly contradict each other”
- passStates at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash.1.00weight 0.13
met, backed by verified evidence E5, E7, E8. “Use a unique random salt of at least 32 bits per password.”
Penalties
No penalties were applied.
Scorer metadata
- corpus
- {"version":2,"snapshot":"2026-10-01","hash":"e12dc326c69a32d91f85f6355dc63e2a7d672b605685a1ffa1582a42bb1c561e","documents":5}
- opened documents
- ["owasp-password-storage","nist-sp800-63b-4"]
- opened by role
- {"primary":["owasp-password-storage","nist-sp800-63b-4"]}
- displayed share
- {"owasp-password-storage":1,"nist-sp800-63b-4":0.0964,"nist-sp800-63b-3":0.0014,"rfc9106":0.009}
- cited count
- 2
- verified count
- 2
- hallucinated count
- 0
- citation integrity
- 1
- hallucinated urls
- []
- evidence block
- true
- evidence block repaired
- false
- evidence count
- 12
- verified quote count
- 12
- verified quote rate
- 1
- evidence
- [{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP's default recommendation is Argon2id with a minimum configuration of 19 MiB memory, 2 iterations, 1 degree of parallelism.","quote":"Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP says to use PBKDF2 with HMAC-SHA-256 and at least 600,000 iterations when FIPS-140 compliance is required.","quote":"If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP says PBKDF2 should be the preferred algorithm when FIPS-validated implementations are required, and asserts that NIST SP 800-63B-4 permits it.","quote":"Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP recommends bcrypt only for legacy systems where Argon2 and scrypt are not available.","quote":"The bcrypt password hashing function should only be used for password storage in legacy systems where Argon2 and scrypt are not available.","verified":true,"document":"owasp-password-storage"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST SP 800-63B-4 requires passwords to be salted and hashed with a suitable password hashing scheme.","quote":"Passwords SHALL be salted and hashed using a suitable password hashing scheme.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST points to SP 800-132 or updated NIST guidance for the approved scheme, without naming an algorithm.","quote":"An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST says the cost factor should be as high as practical.","quote":"The chosen cost factor SHOULD be as high as practical without negatively impacting verifier performance.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST requires a salt of at least 32 bits.","quote":"The salt SHALL be at least 32 bits in length and chosen to minimize salt value collisions among stored hashes","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST recommends storing the scheme and cost factor with each password to allow migration.","quote":"A reference to the password hashing scheme used, including the cost factor, SHOULD be stored for each password to allow migration to new algorithms and work factors.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST recommends an additional keyed hash or encryption step with a secret key known only to the verifier (equivalent to OWASP's pepper).","quote":"verifiers SHOULD perform an additional iteration of a keyed hashing or encryption operation using a secret key known only to the verifier.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"For federal-agency verifiers at AAL1, cryptography must be validated to FIPS 140 Level 1.","quote":"Cryptography used by verifiers operated by or on behalf of federal agencies at AAL1 SHALL be validated to meet the requirements of [FIPS140] Level 1.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST says FIPS 140 requirements are satisfied by implementations validated under the CMVP, so validation attaches to the module, not just the algorithm.","quote":"Throughout this document, [FIPS140] requirements are satisfied by security technologies, products, and services that utilize implementations of cryptography validated by the Cryptography Module Validation Program [CMVP].","verified":true,"document":"nist-sp800-63b-4"}]
- judge unverified quotes
- 0
- judge verdicts
- [{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2","E4"]},"nist-current":{"status":"met","support":["E5","E6"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E5","E7","E8"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2","E4"]},"nist-current":{"status":"met","support":["E5","E6","E7"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E5","E7","E8","E9"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2","E4"]},"nist-current":{"status":"met","support":["E5","E6","E7"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E5","E8","E9"]}}]
- judge failures
- 0
- grounded weight share
- 0.875 (88%)
Final answer
Recommendation: If FIPS applies, use PBKDF2-HMAC-SHA-256 at 600,000 or more iterations, run inside a FIPS 140-validated cryptographic module. Add the other SP 800-63B-4 storage controls described below. Use Argon2id, which is OWASP's first choice, only for parts of the service outside the FIPS boundary, or if FIPS turns out not to be required.
Scope of what I checked. I read the OWASP Password Storage Cheat Sheet and the SP 800-63B-4 text on the NIST site. The NIST page is dated 26 Aug 2025 and is the current revision. I did not open SP 800-132, SP 800-131A or any CMVP material. Check the NIST site for anything newer before you rely on this.
What OWASP says (https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html)
- The default is Argon2id with a minimum of 19 MiB of memory, 2 iterations and 1 degree of parallelism. The parameter list also gives equivalent trade-offs such as m=47104 with t=1 and m=12288 with t=3.
- If Argon2id is unavailable, use scrypt. Use bcrypt only for legacy systems.
- FIPS is an explicit exception: "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256."
- The PBKDF2 section says it "should be the preferred algorithm when these are required". It lists 600,000 iterations for HMAC-SHA256 and 220,000 for HMAC-SHA512. It lists 1,400,000 for HMAC-SHA1, for legacy use only.
- OWASP suggests a pepper as defense in depth. It also suggests storing the algorithm and work factor with each hash (for example in PHC string format) to ease migration. Its rule of thumb is that a hash should take under about one second.
What NIST SP 800-63B-4 says (https://pages.nist.gov/800-63-4/sp800-63b.html)
- Passwords "SHALL be salted and hashed using a suitable password hashing scheme."
- The salt must be at least 32 bits and chosen to minimize collisions among stored hashes. Both the salt and the hash must be stored.
- The cost factor SHOULD be as high as practical without hurting verifier performance, and should be increased over time.
- A reference to the scheme and its cost factor SHOULD be stored with each password, to allow migration.
- An approved scheme from the latest SP 800-132, or from updated NIST password-hashing guidance, SHOULD be used.
- Verifiers SHOULD add a keyed hash or encryption step. The key must come from an approved random bit generator and be stored separately from the hashes, ideally in an HSM or TEE. This corresponds to OWASP's pepper.
- On FIPS, cryptography used by verifiers operated by or on behalf of federal agencies must be validated to FIPS 140 Level 1 at AAL1 and AAL2. At AAL3 it must be Level 1 or higher.
- NIST says FIPS 140 requirements are satisfied by implementations validated under the Cryptography Module Validation Program (CMVP). So validation attaches to the module, not just the algorithm.
Where the sources differ or are silent
- NIST names no algorithm in the text I read, and a search of the page found no "PBKDF". It points to SP 800-132, which is the PBKDF2 recommendation. It gives no iteration count, so 600,000 comes from OWASP alone.
- OWASP says PBKDF2 "is permitted by NIST SP 800-63B-4". That is OWASP's reading, bridged through SP 800-132. It is not wording in 800-63B-4.
- OWASP's Argon2id preference is general best practice, not a NIST requirement. It conflicts with FIPS only where validated modules are mandatory. The two sources do not truly contradict each other: OWASP ranks algorithms by attack resistance, and NIST sets requirements on how hashes are built, stored and validated.
How to reconcile them
- Decide whether FIPS applies. NIST's FIPS 140 requirement covers verifiers operated by or on behalf of federal agencies. If the team is a federal contractor, or the contract mandates FIPS, treat it as binding. If the team only wants alignment, Argon2id could still apply. That is a judgment call that neither source states.
- If FIPS applies, use PBKDF2-HMAC-SHA-256 at 600,000 or more iterations inside a CMVP-validated module, such as a validated OpenSSL provider or platform crypto module. Using the PBKDF2 algorithm is not enough on its own.
- Use a unique random salt of at least 32 bits per password. Store the salt, the hash, and the scheme and iteration count.
- Add a secret pepper as a keyed HMAC step, with the key held in an HSM or secrets vault and separate from the database. Both sources recommend this. If the pepper is compromised, users must reset their passwords.
- Benchmark on production hardware. Re-hash at next login when you raise the cost or change algorithm.
- Keep the migration path open. NIST says updated guidance on password hashing schemes may supersede SP 800-132. Have compliance or the assessor confirm the exact FIPS boundary and the approved-algorithm list.
{"evidence": [
{"claim": "OWASP's default recommendation is Argon2id with a minimum configuration of 19 MiB memory, 2 iterations, 1 degree of parallelism.", "quote": "Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP says to use PBKDF2 with HMAC-SHA-256 and at least 600,000 iterations when FIPS-140 compliance is required.", "quote": "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP says PBKDF2 should be the preferred algorithm when FIPS-validated implementations are required, and asserts that NIST SP 800-63B-4 permits it.", "quote": "Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP recommends bcrypt only for legacy systems where Argon2 and scrypt are not available.", "quote": "The bcrypt password hashing function should only be used for password storage in legacy systems where Argon2 and scrypt are not available.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "NIST SP 800-63B-4 requires passwords to be salted and hashed with a suitable password hashing scheme.", "quote": "Passwords SHALL be salted and hashed using a suitable password hashing scheme.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST points to SP 800-132 or updated NIST guidance for the approved scheme, without naming an algorithm.", "quote": "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST says the cost factor should be as high as practical.", "quote": "The chosen cost factor SHOULD be as high as practical without negatively impacting verifier performance.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST requires a salt of at least 32 bits.", "quote": "The salt SHALL be at least 32 bits in length and chosen to minimize salt value collisions among stored hashes", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST recommends storing the scheme and cost factor with each password to allow migration.", "quote": "A reference to the password hashing scheme used, including the cost factor, SHOULD be stored for each password to allow migration to new algorithms and work factors.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST recommends an additional keyed hash or encryption step with a secret key known only to the verifier (equivalent to OWASP's pepper).", "quote": "verifiers SHOULD perform an additional iteration of a keyed hashing or encryption operation using a secret key known only to the verifier.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "For federal-agency verifiers at AAL1, cryptography must be validated to FIPS 140 Level 1.", "quote": "Cryptography used by verifiers operated by or on behalf of federal agencies at AAL1 SHALL be validated to meet the requirements of [FIPS140] Level 1.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST says FIPS 140 requirements are satisfied by implementations validated under the CMVP, so validation attaches to the module, not just the algorithm.", "quote": "Throughout this document, [FIPS140] requirements are satisfied by security technologies, products, and services that utilize implementations of cryptography validated by the Cryptography Module Validation Program [CMVP].", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"}
]}Provenance of the numbers on this page
- run set
- scaffold-or-scale-v1
- models
- Haiku 4.5, Sonnet 5.5, Opus 5.5
- effort
- high
- repeats
- 2
- fixtures
- replay
- runs
- 900 / 900
- strategy cost
- $354.44
- judge cost
- $9.19
- wall clock
- 9h 48m
- git
- fce5807b2d
Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.