RunHaiku 4.5repeat 2
Plan-execute on OWASP vs NIST on password hashing
claude-haiku-4-5-20251001__plan-execute__research-password-hashing__r2
Score
0.938
- Cost
- $0.185
- + $0.081 judging
- Latency
- 1m 54s
- setup 0.0s, grading 6.0s
- Turns
- 36
- Tool calls
- 32
- Subagents
- 0
- Tokens in / out
- 592k / 9.7k
- 545,716 read from cache
Did the design run?
- Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
- No API failures: heldharness.apiNo model call failed at the API.
- Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
- No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
- Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
- No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
- Builtin tools: heldharness.builtin-toolsA check recorded with the run.
- Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
- No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
- Judge answered: heldharness.judgeAt least one grading call returned a usable verdict.
- The plan was a plan: heldplan-execute.planThe planner’s reply was read through its schema and held at least one step.
- Each call was shown the right tools: heldplan-execute.toolsThe planner was shown no tools and the executor the task’s full toolset.
Phases
Drawn from the phase records: timing and cost per call, no tool calls or messages.
| Phase | Duration | Cost | Turns | Tools | Model |
|---|---|---|---|---|---|
| Plan | 16.2s | $0.014 | 3 | 0 | claude-haiku-4-5-20251001 |
| Execute | 1m 38s | $0.171 | 33 | 32 | claude-haiku-4-5-20251001 |
The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.
Runs you can replay
- A gate that blocked the right moveGated-execute · Haiku 4.5
- Three proposals beat one agentPropose-select · Haiku 4.5
- A verifier that earned its keepPlan-execute-verify · Sonnet 5.5
- A revision that did not helpPlan-execute-verify · Haiku 4.5
- The control, missing a stepSingle · Haiku 4.5
Record this cell yourself: npm run bench -- run --strategies plan-execute --tasks research-password-hashing --models claude-haiku-4-5-20251001 --repeats 1
Score breakdown
- passStates that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism.1.00weight 0.19
met, backed by verified evidence E1. “Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.”
- passStates OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required.1.00weight 0.13
met, backed by verified evidence E2. “If Argon2id is unavailable, OWASP recommends scrypt as an alternative, and for legacy systems, bcrypt with a work factor of 10 or more.”
- passStates that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used.1.00weight 0.19
met, backed by verified evidence E4, E5, E6. “Passwords SHALL be salted and hashed using a suitable password hashing scheme" and that "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.”
- partialStates that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording.0.50weight 0.13
partial, backed by verified evidence E7, E8. “the earlier NIST SP 800-63B Revision 3 was more progressive, noting that "Examples of suitable key derivation functions include Password-based Key Derivation Function 2 (PBKDF2) [SP 800-132] and Balloon [BALLOON]"”
- passExplains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply.1.00weight 0.25
met, backed by verified evidence E2, E3. “Teams must use **PBKDF2 with HMAC-SHA-256 and at least 600,000 iterations**, as this is the only algorithm with FIPS-140 validated implementations.”
- passStates at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash.1.00weight 0.13
met, backed by verified evidence E6. “The recommendations align on the principle that password hashing must be slow and computationally expensive”
Penalties
No penalties were applied.
Scorer metadata
- corpus
- {"version":2,"snapshot":"2026-10-01","hash":"e12dc326c69a32d91f85f6355dc63e2a7d672b605685a1ffa1582a42bb1c561e","documents":5}
- opened documents
- ["owasp-password-storage","nist-sp800-63b-4","nist-sp800-63b-3"]
- opened by role
- {"primary":["owasp-password-storage","nist-sp800-63b-4"],"superseded":["nist-sp800-63b-3"]}
- displayed share
- {"owasp-password-storage":1,"owasp-authentication":0.0556,"nist-sp800-63b-3":0.0565,"nist-sp800-63b-4":0.1004,"rfc9106":0.0604}
- cited count
- 3
- verified count
- 3
- hallucinated count
- 0
- citation integrity
- 1
- hallucinated urls
- []
- evidence block
- true
- evidence block repaired
- false
- evidence count
- 8
- verified quote count
- 8
- verified quote rate
- 1
- evidence
- [{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP recommends Argon2id with minimum 19 MiB memory, 2 iterations, and 1 degree of parallelism as the primary password hashing algorithm","quote":"Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP specifies PBKDF2 with 600,000 iterations and HMAC-SHA-256 for FIPS-140 compliance","quote":"If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP states that PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations","quote":"Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.","verified":true,"document":"owasp-password-storage"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST SP 800-63B-4 requires passwords to be salted and hashed using a suitable password hashing scheme","quote":"Passwords SHALL be salted and hashed using a suitable password hashing scheme.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST SP 800-63B-4 references approved schemes from SP 800-132 or updated NIST guidelines","quote":"An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST SP 800-63B-4 requires cost factor to be as high as practical and increased over time","quote":"The chosen cost factor SHOULD be as high as practical without negatively impacting verifier performance. It SHOULD be increased over time to account for increases in computing performance.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-3/sp800-63b.html","claim":"NIST SP 800-63B Rev 3 lists PBKDF2 and Balloon as suitable key derivation functions","quote":"Examples of suitable key derivation functions include Password-based Key Derivation Function 2 (PBKDF2) [SP 800-132] and Balloon [BALLOON].","verified":true,"document":"nist-sp800-63b-3"},{"url":"https://pages.nist.gov/800-63-3/sp800-63b.html","claim":"NIST SP 800-63B Rev 3 emphasizes that memory-hard functions should be used","quote":"A memory-hard function SHOULD be used because it increases the cost of an attack.","verified":true,"document":"nist-sp800-63b-3"}]
- judge unverified quotes
- 0
- judge verdicts
- [{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2"]},"nist-current":{"status":"met","support":["E4","E5","E6"]},"nist-revision":{"status":"partial","support":["E7","E8"]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E6"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2"]},"nist-current":{"status":"met","support":["E4","E5","E6"]},"nist-revision":{"status":"partial","support":["E7","E8"]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E2","E6"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2"]},"nist-current":{"status":"met","support":["E4","E5","E6"]},"nist-revision":{"status":"partial","support":["E7","E8"]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E6"]}}]
- judge failures
- 0
- grounded weight share
- 1
Final answer
Now let me compile a comprehensive answer based on the authoritative sources I've researched.
Answer
OWASP Password Storage Guidance:
OWASP's Password Storage Cheat Sheet recommends different algorithms based on your compliance requirements:
For applications without FIPS requirements, OWASP recommends Argon2id as the primary choice for new systems. The cheat sheet specifies that organizations should "Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism." If Argon2id is unavailable, OWASP recommends scrypt as an alternative, and for legacy systems, bcrypt with a work factor of 10 or more.
However, when FIPS-140 compliance is required, OWASP explicitly states: "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256." Additionally, OWASP notes that "Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required."
NIST SP 800-63B Requirements:
NIST's current guidance in SP 800-63B-4 states that "Passwords SHALL be salted and hashed using a suitable password hashing scheme" and that "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used." The cost factor "SHOULD be as high as practical without negatively impacting verifier performance" and "SHOULD be increased over time to account for increases in computing performance."
NIST SP 800-132 (from 2010), which is the current approved standard referenced, specifies PBKDF2. However, the earlier NIST SP 800-63B Revision 3 was more progressive, noting that "Examples of suitable key derivation functions include Password-based Key Derivation Function 2 (PBKDF2) [SP 800-132] and Balloon [BALLOON]" and importantly stating that "A memory-hard function SHOULD be used because it increases the cost of an attack."
Reconciling the Two for FIPS Compliance:
The key to reconciliation is understanding the different purposes of these documents:
- For federal FIPS-140 compliance: Teams must use PBKDF2 with HMAC-SHA-256 and at least 600,000 iterations, as this is the only algorithm with FIPS-140 validated implementations. This is what both OWASP and NIST effectively require for federal compliance.
- For non-FIPS federal systems: Teams can follow OWASP's recommendation to use Argon2id as the preferred algorithm. While NIST SP 800-63B-4 references "updated NIST guidelines," no official NIST document currently approves Argon2, bcrypt, or scrypt as replacements for the SP 800-132 PBKDF2 standard. However, NIST's own 800-63B Rev 3 acknowledges that memory-hard functions should be used.
- The practical path forward: If federal FIPS compliance is required, use PBKDF2. If not required, teams can use Argon2id following OWASP guidance, which provides better security through memory-hard properties. The recommendations align on the principle that password hashing must be slow and computationally expensive; they differ only on which algorithms meet that requirement under different regulatory constraints.
{"evidence": [{"claim": "OWASP recommends Argon2id with minimum 19 MiB memory, 2 iterations, and 1 degree of parallelism as the primary password hashing algorithm", "quote": "Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"}, {"claim": "OWASP specifies PBKDF2 with 600,000 iterations and HMAC-SHA-256 for FIPS-140 compliance", "quote": "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"}, {"claim": "OWASP states that PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations", "quote": "Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"}, {"claim": "NIST SP 800-63B-4 requires passwords to be salted and hashed using a suitable password hashing scheme", "quote": "Passwords SHALL be salted and hashed using a suitable password hashing scheme.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"}, {"claim": "NIST SP 800-63B-4 references approved schemes from SP 800-132 or updated NIST guidelines", "quote": "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"}, {"claim": "NIST SP 800-63B-4 requires cost factor to be as high as practical and increased over time", "quote": "The chosen cost factor SHOULD be as high as practical without negatively impacting verifier performance. It SHOULD be increased over time to account for increases in computing performance.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"}, {"claim": "NIST SP 800-63B Rev 3 lists PBKDF2 and Balloon as suitable key derivation functions", "quote": "Examples of suitable key derivation functions include Password-based Key Derivation Function 2 (PBKDF2) [SP 800-132] and Balloon [BALLOON].", "url": "https://pages.nist.gov/800-63-3/sp800-63b.html"}, {"claim": "NIST SP 800-63B Rev 3 emphasizes that memory-hard functions should be used", "quote": "A memory-hard function SHOULD be used because it increases the cost of an attack.", "url": "https://pages.nist.gov/800-63-3/sp800-63b.html"}]}Provenance of the numbers on this page
- run set
- scaffold-or-scale-v1
- models
- Haiku 4.5, Sonnet 5.5, Opus 5.5
- effort
- high
- repeats
- 2
- fixtures
- replay
- runs
- 900 / 900
- strategy cost
- $354.44
- judge cost
- $9.19
- wall clock
- 9h 48m
- git
- fce5807b2d
Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.