RunSonnet 5.5repeat 2
Plan-execute on OWASP vs NIST on password hashing
claude-sonnet-5-5__plan-execute__research-password-hashing__r2
Score
0.938
- Cost
- $0.164
- + $0.086 judging
- Latency
- 46.1s
- setup 0.0s, grading 5.2s
- Turns
- 15
- Tool calls
- 12
- Subagents
- 0
- Tokens in / out
- 101k / 6.0k
- 76,498 read from cache
Did the design run?
- Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
- No API failures: heldharness.apiNo model call failed at the API.
- Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
- No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
- Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
- No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
- Builtin tools: heldharness.builtin-toolsA check recorded with the run.
- Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
- No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
- Judge answered: heldharness.judgeAt least one grading call returned a usable verdict.
- The plan was a plan: heldplan-execute.planThe planner’s reply was read through its schema and held at least one step.
- Each call was shown the right tools: heldplan-execute.toolsThe planner was shown no tools and the executor the task’s full toolset.
Phases
Drawn from the phase records: timing and cost per call, no tool calls or messages.
| Phase | Duration | Cost | Turns | Tools | Model |
|---|---|---|---|---|---|
| Plan | 13.4s | $0.020 | 2 | 0 | claude-sonnet-5-5 |
| Execute | 32.7s | $0.144 | 13 | 12 | claude-sonnet-5-5 |
The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.
Runs you can replay
- A gate that blocked the right moveGated-execute · Haiku 4.5
- Three proposals beat one agentPropose-select · Haiku 4.5
- A verifier that earned its keepPlan-execute-verify · Sonnet 5.5
- A revision that did not helpPlan-execute-verify · Haiku 4.5
- The control, missing a stepSingle · Haiku 4.5
Record this cell yourself: npm run bench -- run --strategies plan-execute --tasks research-password-hashing --models claude-sonnet-5-5 --repeats 1
Score breakdown
- passStates that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism.1.00weight 0.19
met, backed by verified evidence E1. “Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.”
- passStates OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required.1.00weight 0.13
met, backed by verified evidence E2, E4. “If Argon2id isn't available, use scrypt (minimum CPU/memory cost 2^17, block size 8, parallelization 1).”
- passStates that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used.1.00weight 0.19
met, backed by verified evidence E6, E7, E9. “An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.”
- partialStates that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording.0.50weight 0.13
partial, backed by verified evidence E10. “It listed PBKDF2 and Balloon as examples of suitable key derivation functions.”
- passExplains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply.1.00weight 0.25
met, backed by verified evidence E2, E3. “If the service must satisfy FIPS, use PBKDF2-HMAC-SHA-256 at OWASP's 600,000 iterations or more.”
- passStates at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash.1.00weight 0.13
met, backed by verified evidence E8, E9. “Both revisions require a salt of at least 32 bits.”
Penalties
No penalties were applied.
Scorer metadata
- corpus
- {"version":2,"snapshot":"2026-10-01","hash":"e12dc326c69a32d91f85f6355dc63e2a7d672b605685a1ffa1582a42bb1c561e","documents":5}
- opened documents
- ["owasp-password-storage","nist-sp800-63b-4","nist-sp800-63b-3"]
- opened by role
- {"primary":["owasp-password-storage","nist-sp800-63b-4"],"superseded":["nist-sp800-63b-3"]}
- displayed share
- {"owasp-password-storage":0.778,"nist-sp800-63b-4":0.0611,"nist-sp800-63b-3":0.0487,"rfc9106":0.0166,"owasp-authentication":0.0142}
- cited count
- 3
- verified count
- 3
- hallucinated count
- 0
- citation integrity
- 1
- hallucinated urls
- []
- evidence block
- true
- evidence block repaired
- false
- evidence count
- 10
- verified quote count
- 10
- verified quote rate
- 1
- evidence
- [{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP's first-choice recommendation is Argon2id with a stated minimum configuration.","quote":"Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP's FIPS-140 fallback is PBKDF2 with HMAC-SHA-256 and at least 600,000 iterations.","quote":"If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP ties PBKDF2 to NIST SP 800-63B-4 and to FIPS-140 validated implementations.","quote":"Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP limits bcrypt to legacy use.","quote":"The bcrypt password hashing function should only be used for password storage in legacy systems where Argon2 and scrypt are not available.","verified":true,"document":"owasp-password-storage"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"SP 800-63B-4 supersedes the earlier SP 800-63B (Rev 3).","quote":"This publication supersedes NIST Special Publication (SP) 800-63B.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST 800-63B-4 requires passwords to be salted and hashed with a suitable password hashing scheme.","quote":"Passwords SHALL be salted and hashed using a suitable password hashing scheme.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST 800-63B-4 only says an approved scheme from SP 800-132 SHOULD be used, and does not name specific algorithms.","quote":"An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST 800-63B-4 sets a minimum salt length of 32 bits and requires storing salt and hash.","quote":"The salt SHALL be at least 32 bits in length and chosen to minimize salt value collisions among stored hashes","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST 800-63B-4 asks for a cost factor as high as practical, increased over time, without giving a number.","quote":"The chosen cost factor SHOULD be as high as practical without negatively impacting verifier performance.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-3/sp800-63b.html","claim":"Rev 3 named PBKDF2 and Balloon as examples and suggested at least 10,000 iterations, which Rev 4 does not state.","quote":"the iteration count SHOULD be as large as verification server performance will allow, typically at least 10,000 iterations.","verified":true,"document":"nist-sp800-63b-3"}]
- judge unverified quotes
- 0
- judge verdicts
- [{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2","E4"]},"nist-current":{"status":"met","support":["E6","E7","E9"]},"nist-revision":{"status":"partial","support":["E10"]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E8","E9"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2","E4"]},"nist-current":{"status":"met","support":["E6","E7","E9"]},"nist-revision":{"status":"partial","support":["E10"]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E8","E9"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2","E4"]},"nist-current":{"status":"met","support":["E6","E7","E9"]},"nist-revision":{"status":"partial","support":["E10"]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E8","E9"]}}]
- judge failures
- 0
- grounded weight share
- 1
Final answer
Short answer: To meet FIPS, use PBKDF2 with HMAC-SHA-256 at OWASP's stated work factor and a unique salt. If FIPS isn't a constraint, use Argon2id. NIST SP 800-63B-4 names no specific algorithm, so OWASP's FIPS fallback and NIST's requirements don't conflict.
What OWASP recommends
The cheat sheet's summary lists these options in order:
- Argon2id is the first choice. OWASP says to use "Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism."
- If Argon2id isn't available, use scrypt (minimum CPU/memory cost 2^17, block size 8, parallelization 1).
- bcrypt is for legacy systems only, with a work factor of 10 or more and a 72-byte password limit.
- For FIPS, it says "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256."
The PBKDF2 section adds that it "is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required."
Its iteration counts are:
- PBKDF2-HMAC-SHA256: 600,000, which it marks as recommended.
- PBKDF2-HMAC-SHA512: 220,000.
- PBKDF2-HMAC-SHA1: 1,400,000, which it marks as legacy only.
OWASP also recommends salting and, optionally, a pepper. It says calculating a hash "should take less than one second." These numbers are from the page as I read it today (2026-10-13), and OWASP says work factors should be raised over time.
What NIST SP 800-63B requires
The current revision is SP 800-63B-4. The page I opened is dated 26 Aug 2025 and says "This publication supersedes NIST Special Publication (SP) 800-63B." That is the Rev 3 document. The requirements for password storage are:
- Verifiers must store passwords in a form resistant to offline attacks. Passwords "SHALL be salted and hashed using a suitable password hashing scheme."
- The salt "SHALL be at least 32 bits in length" and must be chosen to minimize collisions among stored hashes. The salt and the hash must both be stored.
- The cost factor "SHOULD be as high as practical without negatively impacting verifier performance" and should increase over time.
- "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used." This is a SHOULD, not a SHALL.
- Storing the scheme and cost factor with each password is a SHOULD, so you can migrate later.
- Verifiers SHOULD also do an extra keyed hash or encryption step with a secret key held only by the verifier. That key should be stored separately and ideally in an HSM or TEE.
The text I read does not name Argon2, scrypt or bcrypt. It names no minimum iteration count and does not mention FIPS 140. It points to SP 800-132 (a 2010 publication on password-based key derivation) as the source of approved schemes. I could not open SP 800-132 itself: the csrc.nist.gov URL I tried returned a 404. So I am relying on the 800-63B-4 and OWASP pages for what it covers, and I cannot say from a source whether Argon2, scrypt or bcrypt is on any NIST or CMVP approved list.
How Rev 4 differs from Rev 3
Rev 3 (https://pages.nist.gov/800-63-3/sp800-63b.html) was more specific. It listed PBKDF2 and Balloon as examples of suitable key derivation functions. It also said "the iteration count SHOULD be as large as verification server performance will allow, typically at least 10,000 iterations." Rev 4's text has no 10,000 figure. Both revisions require a salt of at least 32 bits.
How to reconcile them
- The two sources don't conflict. OWASP's first choice is Argon2id. NIST's text does not mention Argon2id, and it points to SP 800-132 for approved schemes. OWASP itself handles the gap with its FIPS-140 fallback.
- If the service must satisfy FIPS, use PBKDF2-HMAC-SHA-256 at OWASP's 600,000 iterations or more. Use a unique random salt of at least 32 bits (longer is fine). Store the scheme and cost with each hash, and add a pepper kept in an HSM or a secrets vault.
- If FIPS is not binding, use Argon2id.
- Running PBKDF2 inside a FIPS-validated cryptographic module is my inference. OWASP says only that FIPS-140 validated implementations exist, and I didn't find a NIST statement on it. Confirm the module's validation status and whether your assessor requires it.
- Treat 600,000 as a floor and benchmark it on your own hardware. The 10,000 in Rev 3 is outdated and far below OWASP's figure. Use OWASP's figure, since NIST 800-63B-4 sets no number.
{"evidence": [
{"claim": "OWASP's first-choice recommendation is Argon2id with a stated minimum configuration.", "quote": "Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP's FIPS-140 fallback is PBKDF2 with HMAC-SHA-256 and at least 600,000 iterations.", "quote": "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP ties PBKDF2 to NIST SP 800-63B-4 and to FIPS-140 validated implementations.", "quote": "Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP limits bcrypt to legacy use.", "quote": "The bcrypt password hashing function should only be used for password storage in legacy systems where Argon2 and scrypt are not available.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "SP 800-63B-4 supersedes the earlier SP 800-63B (Rev 3).", "quote": "This publication supersedes NIST Special Publication (SP) 800-63B.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST 800-63B-4 requires passwords to be salted and hashed with a suitable password hashing scheme.", "quote": "Passwords SHALL be salted and hashed using a suitable password hashing scheme.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST 800-63B-4 only says an approved scheme from SP 800-132 SHOULD be used, and does not name specific algorithms.", "quote": "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST 800-63B-4 sets a minimum salt length of 32 bits and requires storing salt and hash.", "quote": "The salt SHALL be at least 32 bits in length and chosen to minimize salt value collisions among stored hashes", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST 800-63B-4 asks for a cost factor as high as practical, increased over time, without giving a number.", "quote": "The chosen cost factor SHOULD be as high as practical without negatively impacting verifier performance.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "Rev 3 named PBKDF2 and Balloon as examples and suggested at least 10,000 iterations, which Rev 4 does not state.", "quote": "the iteration count SHOULD be as large as verification server performance will allow, typically at least 10,000 iterations.", "url": "https://pages.nist.gov/800-63-3/sp800-63b.html"}
]}Provenance of the numbers on this page
- run set
- scaffold-or-scale-v1
- models
- Haiku 4.5, Sonnet 5.5, Opus 5.5
- effort
- high
- repeats
- 2
- fixtures
- replay
- runs
- 900 / 900
- strategy cost
- $354.44
- judge cost
- $9.19
- wall clock
- 9h 48m
- git
- fce5807b2d
Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.