RunSonnet 5.5repeat 2
Plan-execute-verify on OWASP vs NIST on password hashing
claude-sonnet-5-5__plan-execute-verify__research-password-hashing__r2
Score
0.875
- Cost
- $0.249
- + $0.093 judging
- Latency
- 54.5s
- setup 0.0s, grading 5.5s
- Turns
- 20
- Tool calls
- 15
- Subagents
- 0
- Tokens in / out
- 147k / 7.1k
- 101,355 read from cache
Did the design run?
- Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
- No API failures: heldharness.apiNo model call failed at the API.
- Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
- No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
- Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
- No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
- Builtin tools: heldharness.builtin-toolsA check recorded with the run.
- Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
- No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
- Judge answered: heldharness.judgeAt least one grading call returned a usable verdict.
- The plan was a plan: heldplan-execute-verify.planThe planner’s reply was read through its schema and held at least one step.
- Every verdict was read: heldplan-execute-verify.verdictsEvery review round ended with a verdict read from the schema reply, not a default.
- The loop did what its verdicts said: heldplan-execute-verify.loopEvery verdict that called for a revision got one, and nothing was revised without such a verdict.
- Each call was shown the right tools: heldplan-execute-verify.toolsThe planner was shown no tools, the verifier the read-only ones, and the executor and reviser the full toolset.
Phases
Drawn from the phase records: timing and cost per call, no tool calls or messages.
| Phase | Duration | Cost | Turns | Tools | Model |
|---|---|---|---|---|---|
| Plan | 15.4s | $0.020 | 2 | 0 | claude-sonnet-5-5 |
| Execute | 29.4s | $0.150 | 10 | 9 | claude-sonnet-5-5 |
| Verify | 9.8s | $0.080 | 8 | 6 | claude-sonnet-5-5 |
The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.
Runs you can replay
- A gate that blocked the right moveGated-execute · Haiku 4.5
- Three proposals beat one agentPropose-select · Haiku 4.5
- A verifier that earned its keepPlan-execute-verify · Sonnet 5.5
- A revision that did not helpPlan-execute-verify · Haiku 4.5
- The control, missing a stepSingle · Haiku 4.5
Record this cell yourself: npm run bench -- run --strategies plan-execute-verify --tasks research-password-hashing --models claude-sonnet-5-5 --repeats 1
Score breakdown
- passStates that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism.1.00weight 0.19
met, backed by verified evidence E1. “Argon2id is first: "Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism."”
- passStates OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required.1.00weight 0.13
met, backed by verified evidence E2. “scrypt is the fallback if Argon2id is unavailable.”
- passStates that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used.1.00weight 0.19
met, backed by verified evidence E7, E8. “"An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used." This is a SHOULD, not a SHALL.”
- failStates that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording.0.00weight 0.13
Not stated.
- passExplains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply.1.00weight 0.25
met, backed by verified evidence E2, E3. “Use PBKDF2-HMAC-SHA-256 with at least 600,000 iterations, as OWASP recommends.”
- passStates at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash.1.00weight 0.13
met, backed by verified evidence E7, E9, E10, E12. “The sheet recommends a unique random salt per password”
Penalties
No penalties were applied.
Scorer metadata
- corpus
- {"version":2,"snapshot":"2026-10-01","hash":"e12dc326c69a32d91f85f6355dc63e2a7d672b605685a1ffa1582a42bb1c561e","documents":5}
- opened documents
- ["owasp-password-storage","nist-sp800-63b-4"]
- opened by role
- {"primary":["owasp-password-storage","nist-sp800-63b-4"]}
- displayed share
- {"owasp-password-storage":1,"nist-sp800-63b-4":0.0724,"rfc9106":0.0076,"nist-sp800-63b-3":0.0014,"owasp-authentication":0.0068}
- cited count
- 2
- verified count
- 2
- hallucinated count
- 0
- citation integrity
- 1
- hallucinated urls
- []
- evidence block
- true
- evidence block repaired
- false
- evidence count
- 12
- verified quote count
- 12
- verified quote rate
- 1
- evidence
- [{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP's primary recommendation is Argon2id with a minimum configuration.","quote":"Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP recommends PBKDF2 with HMAC-SHA-256 and at least 600,000 iterations when FIPS-140 compliance is required.","quote":"If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP states PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, so it is preferred when FIPS is required.","quote":"Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP gives a lower iteration count for PBKDF2-HMAC-SHA512.","quote":"PBKDF2-HMAC-SHA512: 220,000 iterations","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP advises an optional pepper stored separately from the database.","quote":"Unlike a password salt, the pepper should not be public and should not be stored along with the generated hash.","verified":true,"document":"owasp-password-storage"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST SP 800-63B-4 supersedes the earlier SP 800-63B and is dated 26 Aug 2025.","quote":"This publication supersedes NIST Special Publication (SP) 800-63B.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST requires passwords to be salted and hashed with a suitable password hashing scheme.","quote":"Passwords SHALL be salted and hashed using a suitable password hashing scheme.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST defers to an approved scheme in SP 800-132 or updated NIST guidance, as a SHOULD, without naming an algorithm in this passage.","quote":"An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST requires a salt of at least 32 bits and storage of both salt and hash.","quote":"The salt SHALL be at least 32 bits in length and chosen to minimize salt value collisions among stored hashes","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST recommends storing the scheme and cost factor to allow migration.","quote":"A reference to the password hashing scheme used, including the cost factor, SHOULD be stored for each password to allow migration to new algorithms and work factors.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST recommends an additional keyed hash or encryption step with a secret key stored separately, ideally in an HSM or TEE.","quote":"In addition, verifiers SHOULD perform an additional iteration of a keyed hashing or encryption operation using a secret key known only to the verifier.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST says the cost factor should be as high as practical and increased over time.","quote":"The chosen cost factor SHOULD be as high as practical without negatively impacting verifier performance.","verified":true,"document":"nist-sp800-63b-4"}]
- judge unverified quotes
- 0
- judge verdicts
- [{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2"]},"nist-current":{"status":"met","support":["E7","E8"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E7","E9","E10","E12"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2"]},"nist-current":{"status":"met","support":["E6","E7","E8","E10","E12"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E7","E9","E12"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2"]},"nist-current":{"status":"met","support":["E6","E7","E8","E10","E12"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E7","E9","E12"]}}]
- judge failures
- 0
- grounded weight share
- 0.875 (88%)
Final answer
Short answer: Use PBKDF2-HMAC-SHA-256 with a FIPS 140-validated module. OWASP recommends Argon2id in general but points to PBKDF2 when FIPS compliance is required. The current NIST SP 800-63B (Revision 4) sets requirements on salting, cost factor and storage, and does not name a required algorithm in the passage I read.
What OWASP recommends
The OWASP Password Storage Cheat Sheet is at https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html. It gives an ordered list:
- Argon2id is first: "Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism."
- scrypt is the fallback if Argon2id is unavailable.
- bcrypt is for legacy systems only, with a work factor of 10 or more and a 72-byte password limit.
- For FIPS, it says: "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256."
In its PBKDF2 section, OWASP gives these iteration counts:
- PBKDF2-HMAC-SHA256: 600,000 iterations (recommended).
- PBKDF2-HMAC-SHA512: 220,000 iterations.
- PBKDF2-HMAC-SHA1: 1,400,000 iterations, legacy only.
OWASP also states that "Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required."
The sheet recommends a unique random salt per password, and says libraries usually generate it for you. It also suggests an optional secret pepper, stored separately from the database (for example in a secrets vault or HSM). It recommends storing the algorithm and work factor with each hash so you can migrate later.
What NIST SP 800-63B requires
The page at https://pages.nist.gov/800-63-4/sp800-63b.html is titled "NIST SP 800-63B-4". It is dated 26 Aug 2025, and its abstract says "This publication supersedes NIST Special Publication (SP) 800-63B." So Revision 4 is the current version, and it replaces Revision 3. I did not check any NIST status page beyond this one.
The password-storage requirements in section 3.1.1.2 are:
- "Passwords SHALL be salted and hashed using a suitable password hashing scheme."
- The cost factor SHOULD be as high as practical without hurting verifier performance, and SHOULD be increased over time.
- "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used." This is a SHOULD, not a SHALL.
- "The salt SHALL be at least 32 bits in length", chosen to minimize collisions among stored hashes.
- The salt and hash SHALL both be stored. A reference to the scheme and cost factor SHOULD be stored too, so you can migrate to new algorithms and work factors.
- Verifiers SHOULD add a keyed hash or encryption step using a secret key. That key SHALL come from an approved random bit generator and SHALL be stored separately from the hashes. NIST says it SHOULD be kept in a hardware-protected area such as an HSM or TEE. This is NIST's equivalent of a pepper.
In the passage I read, 800-63B-4 does not name PBKDF2 or Argon2id. It defers to SP 800-132 or future NIST guidance. I did not open SP 800-132, so I cannot confirm from a source I read that PBKDF2 is the scheme it approves. OWASP asserts this ("permitted by NIST SP 800-63B-4"), but it is OWASP's statement, not NIST's. I also did not open any source on whether Argon2id is FIPS-approved, so I make no claim either way.
How to reconcile them (these steps are my inference, not stated by either source)
- Do not choose Argon2id, scrypt or bcrypt as the primary scheme if the system must use FIPS-approved algorithms. OWASP itself points to PBKDF2 for that case, and NIST's "approved scheme" language points toward SP 800-132.
- Use PBKDF2-HMAC-SHA-256 with at least 600,000 iterations, as OWASP recommends. Use a unique random salt per password. 16 bytes is a common choice and comfortably exceeds NIST's 32-bit minimum. Run it in a FIPS 140-validated cryptographic module.
- Add NIST's keyed step: an HMAC with a secret key from an approved random bit generator, stored in an HSM or secrets vault. This also matches OWASP's pepper advice.
- Store the scheme and cost factor with each hash. Re-tune the iteration count as hardware improves, and plan to move to Argon2id if your FIPS scope later allows it. Both sources support this kind of migration.
- Confirm the details with your compliance or authorizing officer. FIPS applicability depends on the system's authorization boundary, and I did not verify that against any source.
{"evidence": [
{"claim": "OWASP's primary recommendation is Argon2id with a minimum configuration.", "quote": "Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP recommends PBKDF2 with HMAC-SHA-256 and at least 600,000 iterations when FIPS-140 compliance is required.", "quote": "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP states PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, so it is preferred when FIPS is required.", "quote": "Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP gives a lower iteration count for PBKDF2-HMAC-SHA512.", "quote": "PBKDF2-HMAC-SHA512: 220,000 iterations", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP advises an optional pepper stored separately from the database.", "quote": "Unlike a password salt, the pepper should not be public and should not be stored along with the generated hash.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "NIST SP 800-63B-4 supersedes the earlier SP 800-63B and is dated 26 Aug 2025.", "quote": "This publication supersedes NIST Special Publication (SP) 800-63B.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST requires passwords to be salted and hashed with a suitable password hashing scheme.", "quote": "Passwords SHALL be salted and hashed using a suitable password hashing scheme.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST defers to an approved scheme in SP 800-132 or updated NIST guidance, as a SHOULD, without naming an algorithm in this passage.", "quote": "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST requires a salt of at least 32 bits and storage of both salt and hash.", "quote": "The salt SHALL be at least 32 bits in length and chosen to minimize salt value collisions among stored hashes", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST recommends storing the scheme and cost factor to allow migration.", "quote": "A reference to the password hashing scheme used, including the cost factor, SHOULD be stored for each password to allow migration to new algorithms and work factors.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST recommends an additional keyed hash or encryption step with a secret key stored separately, ideally in an HSM or TEE.", "quote": "In addition, verifiers SHOULD perform an additional iteration of a keyed hashing or encryption operation using a secret key known only to the verifier.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST says the cost factor should be as high as practical and increased over time.", "quote": "The chosen cost factor SHOULD be as high as practical without negatively impacting verifier performance.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"}
]}Provenance of the numbers on this page
- run set
- scaffold-or-scale-v1
- models
- Haiku 4.5, Sonnet 5.5, Opus 5.5
- effort
- high
- repeats
- 2
- fixtures
- replay
- runs
- 900 / 900
- strategy cost
- $354.44
- judge cost
- $9.19
- wall clock
- 9h 48m
- git
- fce5807b2d
Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.