RunOpus 5.5repeat 1
Plan-execute on OWASP vs NIST on password hashing
claude-opus-5-5__plan-execute__research-password-hashing__r1
Score
1.000
- Cost
- $0.509
- + $0.113 judging
- Latency
- 1m 40s
- setup 0.0s, grading 5.1s
- Turns
- 28
- Tool calls
- 25
- Subagents
- 0
- Tokens in / out
- 222k / 10k
- 186,124 read from cache
Did the design run?
- Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
- No API failures: heldharness.apiNo model call failed at the API.
- Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
- No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
- Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
- No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
- Builtin tools: heldharness.builtin-toolsA check recorded with the run.
- Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
- No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
- Judge answered: heldharness.judgeAt least one grading call returned a usable verdict.
- The plan was a plan: heldplan-execute.planThe planner’s reply was read through its schema and held at least one step.
- Each call was shown the right tools: heldplan-execute.toolsThe planner was shown no tools and the executor the task’s full toolset.
Phases
Drawn from the phase records: timing and cost per call, no tool calls or messages.
| Phase | Duration | Cost | Turns | Tools | Model |
|---|---|---|---|---|---|
| Plan | 22.1s | $0.051 | 2 | 0 | claude-opus-5-5 |
| Execute | 1m 18s | $0.458 | 26 | 25 | claude-opus-5-5 |
The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.
Runs you can replay
- A gate that blocked the right moveGated-execute · Haiku 4.5
- Three proposals beat one agentPropose-select · Haiku 4.5
- A verifier that earned its keepPlan-execute-verify · Sonnet 5.5
- A revision that did not helpPlan-execute-verify · Haiku 4.5
- The control, missing a stepSingle · Haiku 4.5
Record this cell yourself: npm run bench -- run --strategies plan-execute --tasks research-password-hashing --models claude-opus-5-5 --repeats 1
Score breakdown
- passStates that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism.1.00weight 0.19
met, backed by verified evidence E1. “"Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism."”
- passStates OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required.1.00weight 0.13
met, backed by verified evidence E2, E4. “If Argon2id is unavailable, use scrypt (N=2^17, r=8, p=1). bcrypt is for legacy systems only”
- passStates that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used.1.00weight 0.19
met, backed by verified evidence E8, E9, E10. “"An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used."”
- passStates that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording.1.00weight 0.13
met, backed by verified evidence E16. “Rev 3 named PBKDF2 and Balloon as examples and said "A memory-hard function SHOULD be used."”
- passExplains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply.1.00weight 0.25
met, backed by verified evidence E2, E3. “The two sources don't conflict. OWASP ranks Argon2id first in general but names PBKDF2 for FIPS cases.”
- passStates at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash.1.00weight 0.13
met, backed by verified evidence E8, E11. “a unique salt of at least 32 bits;”
Penalties
No penalties were applied.
Scorer metadata
- corpus
- {"version":2,"snapshot":"2026-10-01","hash":"e12dc326c69a32d91f85f6355dc63e2a7d672b605685a1ffa1582a42bb1c561e","documents":5}
- opened documents
- ["owasp-password-storage","nist-sp800-63b-4","nist-sp800-63b-3"]
- opened by role
- {"primary":["owasp-password-storage","nist-sp800-63b-4"],"superseded":["nist-sp800-63b-3"]}
- displayed share
- {"owasp-password-storage":0.8205,"rfc9106":0.0166,"nist-sp800-63b-4":0.097,"nist-sp800-63b-3":0.0562,"owasp-authentication":0.0142}
- cited count
- 3
- verified count
- 3
- hallucinated count
- 0
- citation integrity
- 1
- hallucinated urls
- []
- evidence block
- true
- evidence block repaired
- false
- evidence count
- 17
- verified quote count
- 17
- verified quote rate
- 1
- evidence
- [{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP's default recommendation is Argon2id with at least 19 MiB memory, 2 iterations, parallelism 1.","quote":"Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP recommends PBKDF2-HMAC-SHA-256 with 600,000+ iterations when FIPS-140 compliance is required.","quote":"If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP says PBKDF2 should be preferred where NIST SP 800-63B-4 / FIPS-140 apply.","quote":"Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP restricts bcrypt to legacy systems.","quote":"The bcrypt password hashing function should only be used for password storage in legacy systems where Argon2 and scrypt are not available.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP says peppers must be stored separately from the password database.","quote":"The pepper should be stored separately from the password database.","verified":true,"document":"owasp-password-storage"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"SP 800-63B-4 supersedes the earlier SP 800-63B.","quote":"This publication supersedes NIST Special Publication (SP) 800-63B.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-3/sp800-63b.html","claim":"Rev 3 was superseded by SP 800-63-4 as of August 1, 2025.","quote":"This revision of NIST SP 800-63 has been superseded by NIST SP 800-63-4 as of August 1, 2025.","verified":true,"document":"nist-sp800-63b-3"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"SP 800-63B-4 requires passwords to be salted and hashed with a suitable password hashing scheme.","quote":"Passwords SHALL be salted and hashed using a suitable password hashing scheme.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"SP 800-63B-4 says the cost factor should be as high as practical.","quote":"The chosen cost factor SHOULD be as high as practical without negatively impacting verifier performance.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"SP 800-63B-4 defers algorithm choice to SP 800-132 or later NIST guidance on approved password hashing schemes.","quote":"An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"SP 800-63B-4 requires a salt of at least 32 bits.","quote":"The salt SHALL be at least 32 bits in length and chosen to minimize salt value collisions among stored hashes (i.e., to prevent multiple subscriber accounts from having the same hashed password).","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"SP 800-63B-4 recommends storing the scheme and cost factor so passwords can be migrated.","quote":"A reference to the password hashing scheme used, including the cost factor, SHOULD be stored for each password to allow migration to new algorithms and work factors.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"SP 800-63B-4 recommends an additional keyed hash or encryption step with a verifier-only secret key.","quote":"In addition, verifiers SHOULD perform an additional iteration of a keyed hashing or encryption operation using a secret key known only to the verifier.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"SP 800-63B-4 says the secret key should be kept in a hardware-protected area such as an HSM or TEE.","quote":"It SHOULD be stored and used within a hardware-protected area, such as a hardware security module or trusted execution environment (TEE), such as a trusted platform module (TPM).","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"Federal verifiers at AAL2 must use FIPS 140 Level 1 validated cryptography.","quote":"Cryptography used by verifiers operated by or on behalf of federal agencies at AAL2 SHALL be validated to meet the requirements of [FIPS140] Level 1 unless otherwise specified.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-3/sp800-63b.html","claim":"Superseded Rev 3 recommended memory-hard functions.","quote":"A memory-hard function SHOULD be used because it increases the cost of an attack.","verified":true,"document":"nist-sp800-63b-3"},{"url":"https://pages.nist.gov/800-63-3/sp800-63b.html","claim":"Superseded Rev 3 suggested PBKDF2 iterations of typically at least 10,000.","quote":"Therefore, the iteration count SHOULD be as large as verification server performance will allow, typically at least 10,000 iterations.","verified":true,"document":"nist-sp800-63b-3"}]
- judge unverified quotes
- 0
- judge verdicts
- [{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2","E4"]},"nist-current":{"status":"met","support":["E8","E9","E10"]},"nist-revision":{"status":"met","support":["E16"]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E8","E11"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2","E4"]},"nist-current":{"status":"met","support":["E8","E9","E10"]},"nist-revision":{"status":"met","support":["E16"]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E8","E9","E11"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2","E4"]},"nist-current":{"status":"met","support":["E8","E9","E10"]},"nist-revision":{"status":"met","support":["E16"]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E8","E9","E11"]}}]
- judge failures
- 0
- grounded weight share
- 1
Final answer
For a FIPS-bound service, the setting that meets both sources is PBKDF2-HMAC-SHA-256 with at least 600,000 iterations. It should run inside a FIPS 140-validated cryptographic module, use a unique random salt of at least 32 bits, and ideally add a secret key (a "pepper") held in an HSM. The two sources don't conflict. OWASP ranks Argon2id first in general but names PBKDF2 for FIPS cases. NIST's current revision requires a salted, cost-tuned scheme and points to its own approved-scheme publication rather than naming an algorithm.
1. What OWASP recommends
- Algorithm order: "Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism." If Argon2id is unavailable, use scrypt (N=2^17, r=8, p=1). bcrypt is for legacy systems only, with a work factor of at least 10 and a 72-byte input limit. https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html
- FIPS case: "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256." The alternative with SHA-512 is 220,000 iterations. PBKDF2-HMAC-SHA1 (1,400,000 iterations) is for legacy use only and not for new systems. OWASP explains that PBKDF2 "is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations." https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html
- Salt and pepper: Every password gets a unique salt. A pepper is optional defence in depth. It must be stored separately from the password database, in a secrets vault or HSM. If it is compromised, every affected user has to reset their password. https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html
- Upgrades: Raise the work factor over time and rehash each password when that user next logs in. https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html
2. What the current NIST SP 800-63B requires
- Which revision is current: Revision 4 (SP 800-63B-4). Its page says it "supersedes NIST Special Publication (SP) 800-63B" and carries an August 26, 2025 timestamp (https://pages.nist.gov/800-63-4/sp800-63b.html). The Rev 3 page says Rev 3 was superseded by SP 800-63-4 "as of August 1, 2025" (https://pages.nist.gov/800-63-3/sp800-63b.html). The NIST CSRC publication page returned a 404, so I couldn't check the status there.
Password storage (Sec. 3.1.1.2):
- Passwords must be stored "in a form that is resistant to offline attacks" and "salted and hashed using a suitable password hashing scheme."
- The cost factor "SHOULD be as high as practical" and should rise over time.
- "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used."
- The salt must be at least 32 bits. The salt and hash must both be stored. A record of the scheme and cost factor should be stored to allow migration.
- Verifiers should add a keyed hash or encryption step with a secret key. That key must come from an approved random bit generator and must be stored separately from the hashes, preferably in an HSM or TEE.
- What Rev 4 doesn't say: I searched its text and found no mention of "PBKDF2", "bcrypt", "scrypt", "Balloon" or "memory-hard", and no iteration count. It neither endorses nor bans Argon2, scrypt or bcrypt. It defers to SP 800-132, which it lists as "Recommendation for Password-Based Key Derivation." https://pages.nist.gov/800-63-4/sp800-63b.html
- FIPS 140 (Sec. 2): Cryptography used by verifiers run by or for federal agencies at AAL1 and AAL2 "SHALL be validated to meet the requirements of [FIPS140] Level 1". That validation comes from implementations certified by the Cryptographic Module Validation Program (CMVP). https://pages.nist.gov/800-63-4/sp800-63b.html
- Rev 3, for comparison only (superseded): Rev 3 named PBKDF2 and Balloon as examples and said "A memory-hard function SHOULD be used." It suggested PBKDF2 iterations of "typically at least 10,000." https://pages.nist.gov/800-63-3/sp800-63b.html
3. How to reconcile them
Algorithm: Use PBKDF2-HMAC-SHA-256 (≥600,000 iterations) or PBKDF2-HMAC-SHA-512 (≥220,000), from a FIPS 140-validated module.
- This is OWASP's own FIPS branch.
- It fits NIST's call for an approved scheme from SP 800-132 run in validated cryptography.
NIST minimums:
- a unique salt of at least 32 bits;
- a cost factor set "as high as practical," with OWASP's figures as the floor;
- the algorithm and cost factor stored with each hash, so passwords can be rehashed when the user logs in.
- Pepper: Add a secret-key step, ideally an HMAC with a key from an approved random bit generator, kept in an HSM or TEE. NIST says this SHOULD be done; OWASP treats it as optional. Plan for a pepper rotation forcing password resets, as OWASP warns.
- Argon2id: Keep it for non-FIPS deployments only.
4. Where the sources differ
- How specific they are: OWASP names algorithms and numbers. NIST Rev 4 names neither; it refers to SP 800-132 and only requires a cost factor that is "as high as practical."
- Memory-hard functions: OWASP still puts memory-hard Argon2id and scrypt first. Rev 4 dropped Rev 3's "memory-hard SHOULD" language.
Unconfirmed claims:
- I couldn't open SP 800-132 (404), so I haven't confirmed from NIST's own text which schemes are "approved." Whether Argon2, scrypt or bcrypt are FIPS-approved is unconfirmed by any page I fetched.
- OWASP says HMAC-SHA-256 "is recommended by NIST" and that SP 800-63B-4 permits PBKDF2. Rev 4 itself never names PBKDF2 or HMAC-SHA-256, so those claims rest on its reference to SP 800-132.
- Pepper strength: NIST says verifiers SHOULD add a secret-key step. OWASP only says "consider" one and calls it defence in depth.
{"evidence": [
{"claim": "OWASP's default recommendation is Argon2id with at least 19 MiB memory, 2 iterations, parallelism 1.", "quote": "Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP recommends PBKDF2-HMAC-SHA-256 with 600,000+ iterations when FIPS-140 compliance is required.", "quote": "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP says PBKDF2 should be preferred where NIST SP 800-63B-4 / FIPS-140 apply.", "quote": "Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP restricts bcrypt to legacy systems.", "quote": "The bcrypt password hashing function should only be used for password storage in legacy systems where Argon2 and scrypt are not available.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP says peppers must be stored separately from the password database.", "quote": "The pepper should be stored separately from the password database.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "SP 800-63B-4 supersedes the earlier SP 800-63B.", "quote": "This publication supersedes NIST Special Publication (SP) 800-63B.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "Rev 3 was superseded by SP 800-63-4 as of August 1, 2025.", "quote": "This revision of NIST SP 800-63 has been superseded by NIST SP 800-63-4 as of August 1, 2025.", "url": "https://pages.nist.gov/800-63-3/sp800-63b.html"},
{"claim": "SP 800-63B-4 requires passwords to be salted and hashed with a suitable password hashing scheme.", "quote": "Passwords SHALL be salted and hashed using a suitable password hashing scheme.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "SP 800-63B-4 says the cost factor should be as high as practical.", "quote": "The chosen cost factor SHOULD be as high as practical without negatively impacting verifier performance.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "SP 800-63B-4 defers algorithm choice to SP 800-132 or later NIST guidance on approved password hashing schemes.", "quote": "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "SP 800-63B-4 requires a salt of at least 32 bits.", "quote": "The salt SHALL be at least 32 bits in length and chosen to minimize salt value collisions among stored hashes (i.e., to prevent multiple subscriber accounts from having the same hashed password).", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "SP 800-63B-4 recommends storing the scheme and cost factor so passwords can be migrated.", "quote": "A reference to the password hashing scheme used, including the cost factor, SHOULD be stored for each password to allow migration to new algorithms and work factors.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "SP 800-63B-4 recommends an additional keyed hash or encryption step with a verifier-only secret key.", "quote": "In addition, verifiers SHOULD perform an additional iteration of a keyed hashing or encryption operation using a secret key known only to the verifier.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "SP 800-63B-4 says the secret key should be kept in a hardware-protected area such as an HSM or TEE.", "quote": "It SHOULD be stored and used within a hardware-protected area, such as a hardware security module or trusted execution environment (TEE), such as a trusted platform module (TPM).", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "Federal verifiers at AAL2 must use FIPS 140 Level 1 validated cryptography.", "quote": "Cryptography used by verifiers operated by or on behalf of federal agencies at AAL2 SHALL be validated to meet the requirements of [FIPS140] Level 1 unless otherwise specified.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "Superseded Rev 3 recommended memory-hard functions.", "quote": "A memory-hard function SHOULD be used because it increases the cost of an attack.", "url": "https://pages.nist.gov/800-63-3/sp800-63b.html"},
{"claim": "Superseded Rev 3 suggested PBKDF2 iterations of typically at least 10,000.", "quote": "Therefore, the iteration count SHOULD be as large as verification server performance will allow, typically at least 10,000 iterations.", "url": "https://pages.nist.gov/800-63-3/sp800-63b.html"}
]}Provenance of the numbers on this page
- run set
- scaffold-or-scale-v1
- models
- Haiku 4.5, Sonnet 5.5, Opus 5.5
- effort
- high
- repeats
- 2
- fixtures
- replay
- runs
- 900 / 900
- strategy cost
- $354.44
- judge cost
- $9.19
- wall clock
- 9h 48m
- git
- fce5807b2d
Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.