RunOpus 5.5repeat 2
Plan-execute-verify on OWASP vs NIST on password hashing
claude-opus-5-5__plan-execute-verify__research-password-hashing__r2
Score
0.938
- Cost
- $0.663
- + $0.101 judging
- Latency
- 1m 53s
- setup 0.0s, grading 6.3s
- Turns
- 36
- Tool calls
- 31
- Subagents
- 0
- Tokens in / out
- 249k / 11k
- 191,224 read from cache
Did the design run?
- Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
- No API failures: heldharness.apiNo model call failed at the API.
- Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
- No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
- Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
- No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
- Builtin tools: heldharness.builtin-toolsA check recorded with the run.
- Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
- No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
- Judge answered: heldharness.judgeAt least one grading call returned a usable verdict.
- The plan was a plan: heldplan-execute-verify.planThe planner’s reply was read through its schema and held at least one step.
- Every verdict was read: heldplan-execute-verify.verdictsEvery review round ended with a verdict read from the schema reply, not a default.
- The loop did what its verdicts said: heldplan-execute-verify.loopEvery verdict that called for a revision got one, and nothing was revised without such a verdict.
- Each call was shown the right tools: heldplan-execute-verify.toolsThe planner was shown no tools, the verifier the read-only ones, and the executor and reviser the full toolset.
Phases
Drawn from the phase records: timing and cost per call, no tool calls or messages.
| Phase | Duration | Cost | Turns | Tools | Model |
|---|---|---|---|---|---|
| Plan | 21.1s | $0.043 | 2 | 0 | claude-opus-5-5 |
| Execute | 1m 0s | $0.354 | 19 | 18 | claude-opus-5-5 |
| Verify | 31.4s | $0.266 | 15 | 13 | claude-opus-5-5 |
The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.
Runs you can replay
- A gate that blocked the right moveGated-execute · Haiku 4.5
- Three proposals beat one agentPropose-select · Haiku 4.5
- A verifier that earned its keepPlan-execute-verify · Sonnet 5.5
- A revision that did not helpPlan-execute-verify · Haiku 4.5
- The control, missing a stepSingle · Haiku 4.5
Record this cell yourself: npm run bench -- run --strategies plan-execute-verify --tasks research-password-hashing --models claude-opus-5-5 --repeats 1
Score breakdown
- passStates that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism.1.00weight 0.19
met, backed by verified evidence E1. “**Argon2id**, with at least 19 MiB of memory, 2 iterations and parallelism of 1.”
- passStates OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required.1.00weight 0.13
met, backed by verified evidence E2, E3, E4. “**PBKDF2**, when FIPS-140 compliance is required: at least 600,000 iterations with HMAC-SHA-256.”
- passStates that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used.1.00weight 0.19
met, backed by verified evidence E9, E10, E11. “Salt and hash them "using a suitable password hashing scheme".”
- partialStates that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording.0.50weight 0.13
partial, backed by verified evidence E15. “Rev 3 did name examples: "PBKDF2 [SP 800-132] and Balloon". It also said a memory-hard function SHOULD be used”
- passExplains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply.1.00weight 0.25
met, backed by verified evidence E4, E5, E11. “The PBKDF2 setup above meets every Rev 4 requirement:”
- passStates at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash.1.00weight 0.13
met, backed by verified evidence E9, E10, E12. “A salted password hashing scheme with a cost factor.”
Penalties
No penalties were applied.
Scorer metadata
- corpus
- {"version":2,"snapshot":"2026-10-01","hash":"e12dc326c69a32d91f85f6355dc63e2a7d672b605685a1ffa1582a42bb1c561e","documents":5}
- opened documents
- ["owasp-password-storage","nist-sp800-63b-4","nist-sp800-63b-3"]
- opened by role
- {"primary":["owasp-password-storage","nist-sp800-63b-4"],"superseded":["nist-sp800-63b-3"]}
- displayed share
- {"owasp-password-storage":1,"nist-sp800-63b-4":0.0691,"rfc9106":0.0166,"nist-sp800-63b-3":0.0553,"owasp-authentication":0.021}
- cited count
- 3
- verified count
- 3
- hallucinated count
- 0
- citation integrity
- 1
- hallucinated urls
- []
- evidence block
- true
- evidence block repaired
- false
- evidence count
- 16
- verified quote count
- 16
- verified quote rate
- 1
- evidence
- [{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP recommends Argon2id first with minimum 19 MiB memory, 2 iterations, parallelism 1","quote":"Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP: scrypt if Argon2id is unavailable","quote":"If Argon2id is not available, use scrypt with a minimum CPU/memory cost parameter of (2^17), a minimum block size of 8 (1024 bytes), and a parallelization parameter of 1.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP: bcrypt for legacy systems, work factor 10+, 72-byte limit","quote":"For legacy systems using bcrypt, use a work factor of 10 or more and with a password limit of 72 bytes.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP: PBKDF2-HMAC-SHA-256 with 600,000+ iterations when FIPS-140 is required","quote":"If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP justifies PBKDF2 by NIST SP 800-63B-4 permission and FIPS-140 validated implementations","quote":"Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP: peppers should be stored in secrets vaults or HSMs","quote":"Peppers are secrets and should be stored in \"secrets vaults\" or HSMs (Hardware Security Modules).","verified":true,"document":"owasp-password-storage"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"SP 800-63B-4 supersedes the earlier SP 800-63B","quote":"This publication supersedes NIST Special Publication (SP) 800-63B.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-3/sp800-63b.html","claim":"Rev 3 was superseded by SP 800-63-4 as of August 1, 2025","quote":"This revision of NIST SP 800-63 has been superseded by NIST SP 800-63-4 as of August 1, 2025.","verified":true,"document":"nist-sp800-63b-3"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"Rev 4 requires salted hashing with a suitable password hashing scheme","quote":"Passwords SHALL be salted and hashed using a suitable password hashing scheme.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"Rev 4 sets cost-factor guidance qualitatively, without a number","quote":"The chosen cost factor SHOULD be as high as practical without negatively impacting verifier performance.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"Rev 4 points to approved schemes in SP 800-132 rather than naming an algorithm","quote":"An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"Rev 4 requires a salt of at least 32 bits","quote":"The salt SHALL be at least 32 bits in length and chosen to minimize salt value collisions among stored hashes (i.e., to prevent multiple subscriber accounts from having the same hashed password).","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"Rev 4 recommends an additional keyed hash/encryption step with a verifier-only secret key","quote":"In addition, verifiers SHOULD perform an additional iteration of a keyed hashing or encryption operation using a secret key known only to the verifier.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"Rev 4: the secret key must be stored separately from the hashes","quote":"The secret key value SHALL be stored separately from the hashed passwords.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-3/sp800-63b.html","claim":"Rev 3 (superseded) named PBKDF2 and Balloon as example KDFs","quote":"Examples of suitable key derivation functions include Password-based Key Derivation Function 2 (PBKDF2) [SP 800-132] and Balloon [BALLOON].","verified":true,"document":"nist-sp800-63b-3"},{"url":"https://pages.nist.gov/800-63-3/sp800-63b.html","claim":"Rev 3 (superseded) suggested at least 10,000 PBKDF2 iterations","quote":"Therefore, the iteration count SHOULD be as large as verification server performance will allow, typically at least 10,000 iterations.","verified":true,"document":"nist-sp800-63b-3"}]
- judge unverified quotes
- 0
- judge verdicts
- [{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2","E3","E4"]},"nist-current":{"status":"met","support":["E9","E10","E11"]},"nist-revision":{"status":"partial","support":["E15"]},"reconciliation":{"status":"met","support":["E4","E5","E11"]},"common-ground":{"status":"met","support":["E9","E10","E12"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2","E3","E4"]},"nist-current":{"status":"met","support":["E9","E10","E11"]},"nist-revision":{"status":"partial","support":["E15"]},"reconciliation":{"status":"met","support":["E4","E5","E11"]},"common-ground":{"status":"met","support":["E9","E10"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2","E3","E4"]},"nist-current":{"status":"met","support":["E9","E10","E11"]},"nist-revision":{"status":"partial","support":["E15"]},"reconciliation":{"status":"met","support":["E4","E5","E11"]},"common-ground":{"status":"met","support":["E9","E10"]}}]
- judge failures
- 0
- grounded weight share
- 1
Final answer
Recommendation: use PBKDF2 with HMAC-SHA-256 and at least 600,000 iterations. Give each password a random salt of at least 32 bits. Add a secret-key (pepper) step whose key is kept apart from the hash database, ideally in an HSM. Argon2id is OWASP's first choice only when FIPS doesn't apply.
What OWASP recommends
The OWASP Password Storage Cheat Sheet (https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html) ranks the options in this order:
- Argon2id, with at least 19 MiB of memory, 2 iterations and parallelism of 1. It also lists equivalent settings that trade memory for iterations, from m=47104 (46 MiB), t=1 to m=7168 (7 MiB), t=5.
- scrypt, if Argon2id isn't available: N=2^17 (128 MiB), r=8, p=1, or one of the listed equivalents.
- bcrypt, only for legacy systems: work factor of 10 or more. Input is limited to 72 bytes in most implementations.
- PBKDF2, when FIPS-140 compliance is required: at least 600,000 iterations with HMAC-SHA-256. The PBKDF2 section also gives 220,000 iterations for HMAC-SHA-512. It says HMAC-SHA-1 (1,400,000 iterations) is for legacy systems only.
OWASP explains the FIPS choice by saying PBKDF2 "is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations". It also suggests considering a pepper, stored in a secrets vault or HSM rather than with the hashes.
What the current NIST SP 800-63B requires
The current revision is SP 800-63B-4. The Rev 4 abstract says it "supersedes NIST Special Publication (SP) 800-63B" (https://pages.nist.gov/800-63-4/sp800-63b.html). The Rev 3 page carries a banner saying it was superseded by SP 800-63-4 "as of August 1, 2025" (https://pages.nist.gov/800-63-3/sp800-63b.html). I couldn't open the CSRC landing pages (both returned 404), so that August 1, 2025 date comes from the Rev 3 banner. The Rev 4 HTML page itself only shows a build timestamp of 26 Aug 2025.
Section 3.1.1.2 of Rev 4 requires:
Must (SHALL):
- Store passwords "in a form that is resistant to offline attacks".
- Salt and hash them "using a suitable password hashing scheme".
- Use a salt of at least 32 bits, chosen to minimise collisions.
- Store both the salt and the hash for each password.
Should (SHOULD):
- Set the cost factor as high as practical without hurting verifier performance, and raise it over time.
- Use "an approved password hashing scheme published in the latest revision of [SP800-132]" or later NIST guidance.
- Store a reference to the scheme and cost factor with each password.
- Add a keyed hashing or encryption step using a secret key known only to the verifier. If that step is used, the key must come from an approved random bit generator and must be stored separately from the hashes. It should be kept in hardware such as an HSM or TEE.
Rev 4 does not name any algorithm. A search of the whole page finds no "PBKDF2", and the passage gives no iteration count.
Older context (Rev 3, now superseded)
Rev 3 did name examples: "PBKDF2 [SP 800-132] and Balloon". It also said a memory-hard function SHOULD be used, and that PBKDF2's iteration count should be "typically at least 10,000".
How the two fit together
The PBKDF2 setup above meets every Rev 4 requirement:
- A salted password hashing scheme with a cost factor.
- A salt of at least 32 bits. OWASP sets no salt length, so NIST's minimum is the one to follow.
- The approved-scheme recommendation, through SP 800-132. Rev 4 lists SP 800-132 as "Recommendation for Password-Based Key Derivation", and Rev 3 cited it as the source for PBKDF2. I couldn't open SP 800-132 itself (404), so I haven't quoted it directly.
Where the sources differ:
- Iteration count. OWASP gives a firm minimum of 600,000. Rev 4 gives no number and only says "as high as practical", and the old Rev 3 figure was 10,000. Using OWASP's number is a stricter way to meet NIST's wording; it doesn't conflict with it.
- Ranking. OWASP ranks Argon2id first. Rev 4 only points to approved schemes, and its password storage section doesn't mention Argon2id. That's why OWASP itself switches to PBKDF2 when FIPS applies.
- Pepper. OWASP says to "consider" one. NIST makes it a SHOULD, with SHALL rules if it's used. Under NIST, the team should treat it as expected.
- "Permitted by Rev 4". OWASP says PBKDF2 is permitted by Rev 4. That is an inference through SP 800-132; the Rev 4 text never names PBKDF2.
Neither source says how to get FIPS-140 validation itself. The team should check that its PBKDF2 actually runs inside a validated cryptographic module.
{"evidence": [
{"claim": "OWASP recommends Argon2id first with minimum 19 MiB memory, 2 iterations, parallelism 1", "quote": "Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP: scrypt if Argon2id is unavailable", "quote": "If Argon2id is not available, use scrypt with a minimum CPU/memory cost parameter of (2^17), a minimum block size of 8 (1024 bytes), and a parallelization parameter of 1.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP: bcrypt for legacy systems, work factor 10+, 72-byte limit", "quote": "For legacy systems using bcrypt, use a work factor of 10 or more and with a password limit of 72 bytes.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP: PBKDF2-HMAC-SHA-256 with 600,000+ iterations when FIPS-140 is required", "quote": "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP justifies PBKDF2 by NIST SP 800-63B-4 permission and FIPS-140 validated implementations", "quote": "Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP: peppers should be stored in secrets vaults or HSMs", "quote": "Peppers are secrets and should be stored in \"secrets vaults\" or HSMs (Hardware Security Modules).", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "SP 800-63B-4 supersedes the earlier SP 800-63B", "quote": "This publication supersedes NIST Special Publication (SP) 800-63B.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "Rev 3 was superseded by SP 800-63-4 as of August 1, 2025", "quote": "This revision of NIST SP 800-63 has been superseded by NIST SP 800-63-4 as of August 1, 2025.", "url": "https://pages.nist.gov/800-63-3/sp800-63b.html"},
{"claim": "Rev 4 requires salted hashing with a suitable password hashing scheme", "quote": "Passwords SHALL be salted and hashed using a suitable password hashing scheme.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "Rev 4 sets cost-factor guidance qualitatively, without a number", "quote": "The chosen cost factor SHOULD be as high as practical without negatively impacting verifier performance.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "Rev 4 points to approved schemes in SP 800-132 rather than naming an algorithm", "quote": "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "Rev 4 requires a salt of at least 32 bits", "quote": "The salt SHALL be at least 32 bits in length and chosen to minimize salt value collisions among stored hashes (i.e., to prevent multiple subscriber accounts from having the same hashed password).", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "Rev 4 recommends an additional keyed hash/encryption step with a verifier-only secret key", "quote": "In addition, verifiers SHOULD perform an additional iteration of a keyed hashing or encryption operation using a secret key known only to the verifier.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "Rev 4: the secret key must be stored separately from the hashes", "quote": "The secret key value SHALL be stored separately from the hashed passwords.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "Rev 3 (superseded) named PBKDF2 and Balloon as example KDFs", "quote": "Examples of suitable key derivation functions include Password-based Key Derivation Function 2 (PBKDF2) [SP 800-132] and Balloon [BALLOON].", "url": "https://pages.nist.gov/800-63-3/sp800-63b.html"},
{"claim": "Rev 3 (superseded) suggested at least 10,000 PBKDF2 iterations", "quote": "Therefore, the iteration count SHOULD be as large as verification server performance will allow, typically at least 10,000 iterations.", "url": "https://pages.nist.gov/800-63-3/sp800-63b.html"}
]}Provenance of the numbers on this page
- run set
- scaffold-or-scale-v1
- models
- Haiku 4.5, Sonnet 5.5, Opus 5.5
- effort
- high
- repeats
- 2
- fixtures
- replay
- runs
- 900 / 900
- strategy cost
- $354.44
- judge cost
- $9.19
- wall clock
- 9h 48m
- git
- fce5807b2d
Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.