ScaffoldBench

RunHaiku 4.5repeat 1

Plan-execute on OWASP vs NIST on password hashing

claude-haiku-4-5-20251001__plan-execute__research-password-hashing__r1

Score

0.813

wrong answer
design ran as designed
Cost
$0.168
+ $0.075 judging
Latency
1m 39s
setup 0.0s, grading 5.9s
Turns
32
Tool calls
28
Subagents
0
Tokens in / out
496k / 8.9k
451,231 read from cache

Did the design run?

every check held
  • Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
  • No API failures: heldharness.apiNo model call failed at the API.
  • Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
  • No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
  • Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
  • No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
  • Builtin tools: heldharness.builtin-toolsA check recorded with the run.
  • Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
  • No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
  • Judge answered: heldharness.judgeAt least one grading call returned a usable verdict.
  • The plan was a plan: heldplan-execute.planThe planner’s reply was read through its schema and held at least one step.
  • Each call was shown the right tools: heldplan-execute.toolsThe planner was shown no tools and the executor the task’s full toolset.

Phases

Plan: 0.0s to 20.2s, finished, 0 tool calls, $0.016
Execute: 20.2s to 1m 39s, finished, 28 tool calls, $0.152

Drawn from the phase records: timing and cost per call, no tool calls or messages.

PhaseDurationCostTurnsToolsModel
Plan20.2s$0.01630claude-haiku-4-5-20251001
Execute1m 19s$0.1522928claude-haiku-4-5-20251001

The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.

Runs you can replay

Record this cell yourself: npm run bench -- run --strategies plan-execute --tasks research-password-hashing --models claude-haiku-4-5-20251001 --repeats 1

Score breakdown

  • passStates that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism.
    1.00weight 0.19

    met, backed by verified evidence E1. “The primary recommendation is to use **Argon2id** with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.”

  • passStates OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required.
    1.00weight 0.13

    met, backed by verified evidence E2. “If Argon2id is unavailable, the guidance suggests **scrypt** as an alternative, and acknowledges **bcrypt** for legacy systems.”

  • passStates that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used.
    1.00weight 0.19

    met, backed by verified evidence E4, E5. “requires that passwords be "salted and hashed using a suitable password hashing scheme" and directs implementers to use "an approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing scheme …[truncated 4 chars]

  • failStates that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording.
    0.00weight 0.13

    Not stated.

  • passExplains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply.
    1.00weight 0.25

    met, backed by verified evidence E2, E3. “When they do have FIPS-140 requirements, both OWASP and NIST support PBKDF2 as the appropriate choice.”

  • partialStates at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash.
    0.50weight 0.13

    partial, backed by verified evidence E5. “The guidance emphasizes that the cost factor "SHOULD be as high as practical without negatively impacting verifier performance"”

Penalties

No penalties were applied.

Scorer metadata

corpus
{"version":2,"snapshot":"2026-10-01","hash":"e12dc326c69a32d91f85f6355dc63e2a7d672b605685a1ffa1582a42bb1c561e","documents":5}
opened documents
["owasp-password-storage","nist-sp800-63b-4","nist-sp800-63b-3"]
opened by role
{"primary":["owasp-password-storage","nist-sp800-63b-4"],"superseded":["nist-sp800-63b-3"]}
displayed share
{"owasp-password-storage":1,"owasp-authentication":0.0277,"nist-sp800-63b-3":0.0526,"nist-sp800-63b-4":0.1625,"rfc9106":0.051}
cited count
3
verified count
3
hallucinated count
0
citation integrity
1
hallucinated urls
[]
evidence block
true
evidence block repaired
false
evidence count
7
verified quote count
7
verified quote rate
1
evidence
[{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP's primary password hashing recommendation is Argon2id with specific minimum parameters","quote":"Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP recommends PBKDF2 when FIPS-140 compliance is required, with specific work factor and hash function","quote":"If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP explicitly identifies PBKDF2 as appropriate for FIPS-140 compliance and notes NIST SP 800-63B-4 support","quote":"Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.","verified":true,"document":"owasp-password-storage"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST SP 800-63B-4 requires passwords be hashed with suitable password hashing schemes, referencing NIST SP 800-132 and updated guidelines","quote":"Passwords SHALL be salted and hashed using a suitable password hashing scheme... An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST SP 800-63B-4 recommends increasing the cost factor of password hashing over time","quote":"The chosen cost factor SHOULD be as high as practical without negatively impacting verifier performance. It SHOULD be increased over time to account for increases in computing performance.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-3/sp800-63b.html","claim":"NIST SP 800-63 Revision 3 mentions PBKDF2 and Balloon as suitable key derivation functions, with preference for memory-hard functions","quote":"Examples of suitable key derivation functions include Password-based Key Derivation Function 2 (PBKDF2) [SP 800-132] and Balloon [BALLOON]. A memory-hard function SHOULD be used because it increases the cost of an attack.","verified":true,"document":"nist-sp800-63b-3"},{"url":"https://pages.nist.gov/800-63-3/sp800-63b.html","claim":"NIST SP 800-63 Revision 3 recommends PBKDF2 iteration count of at least 10,000","quote":"For PBKDF2, the cost factor is an iteration count: the more times the PBKDF2 function is iterated, the longer it takes to compute the password hash. Therefore, the iteration count SHOULD be as large as verification server performance will allow, typically at least 10,000 iterations.","verified":true,"document":"nist-sp800-63b-3"}]
judge unverified quotes
0
judge verdicts
[{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2"]},"nist-current":{"status":"met","support":["E4","E5"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"partial","support":["E5"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2"]},"nist-current":{"status":"met","support":["E4","E5"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E2","E3","E4"]},"common-ground":{"status":"partial","support":["E5"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2"]},"nist-current":{"status":"met","support":["E4"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E2","E3","E4"]},"common-ground":{"status":"partial","support":["E5"]}}]
judge failures
0
grounded weight share
0.875 (88%)

Final answer

Perfect! Now I have all the information needed. Let me compile the findings.

Summary

OWASP Password Storage Guidance:

OWASP recommends a tiered approach: The primary recommendation is to use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism. If Argon2id is unavailable, the guidance suggests scrypt as an alternative, and acknowledges bcrypt for legacy systems. However, OWASP explicitly addresses FIPS compliance: when FIPS-140 compliance is required, the guidance specifies using PBKDF2 with a work factor of 600,000 or more and HMAC-SHA-256. OWASP notes that "PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations."

NIST SP 800-63B-4 Requirements:

The current NIST SP 800-63B-4 requires that passwords be "salted and hashed using a suitable password hashing scheme" and directs implementers to use "an approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes." Rather than prescribing specific algorithms, NIST takes a more flexible approach, referencing both legacy NIST SP 800-132 (which covers PBKDF2) and updated NIST guidelines. The guidance emphasizes that the cost factor "SHOULD be as high as practical without negatively impacting verifier performance" and "SHOULD be increased over time to account for increases in computing performance."

Reconciliation of OWASP and NIST:

The two sources can be reconciled by understanding their different scopes: OWASP provides practical implementation guidance with explicit algorithm recommendations prioritized by security strength and modern design, while NIST provides regulatory compliance flexibility. The key reconciliation point is that OWASP explicitly recognizes FIPS-140 as a distinct requirement and provides PBKDF2 as the solution when this compliance is needed. When a team does not have FIPS-140 requirements, they should follow OWASP's preferred modern algorithms (Argon2id or scrypt). When they do have FIPS-140 requirements, both OWASP and NIST support PBKDF2 as the appropriate choice. NIST SP 800-63B-4 permits this flexibility by allowing "approved password hashing schemes," which includes both modern algorithms (increasingly recognized in updated NIST guidelines) and FIPS-validated schemes like PBKDF2.


{"evidence": [{"claim": "OWASP's primary password hashing recommendation is Argon2id with specific minimum parameters", "quote": "Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"}, {"claim": "OWASP recommends PBKDF2 when FIPS-140 compliance is required, with specific work factor and hash function", "quote": "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"}, {"claim": "OWASP explicitly identifies PBKDF2 as appropriate for FIPS-140 compliance and notes NIST SP 800-63B-4 support", "quote": "Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"}, {"claim": "NIST SP 800-63B-4 requires passwords be hashed with suitable password hashing schemes, referencing NIST SP 800-132 and updated guidelines", "quote": "Passwords SHALL be salted and hashed using a suitable password hashing scheme... An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"}, {"claim": "NIST SP 800-63B-4 recommends increasing the cost factor of password hashing over time", "quote": "The chosen cost factor SHOULD be as high as practical without negatively impacting verifier performance. It SHOULD be increased over time to account for increases in computing performance.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"}, {"claim": "NIST SP 800-63 Revision 3 mentions PBKDF2 and Balloon as suitable key derivation functions, with preference for memory-hard functions", "quote": "Examples of suitable key derivation functions include Password-based Key Derivation Function 2 (PBKDF2) [SP 800-132] and Balloon [BALLOON]. A memory-hard function SHOULD be used because it increases the cost of an attack.", "url": "https://pages.nist.gov/800-63-3/sp800-63b.html"}, {"claim": "NIST SP 800-63 Revision 3 recommends PBKDF2 iteration count of at least 10,000", "quote": "For PBKDF2, the cost factor is an iteration count: the more times the PBKDF2 function is iterated, the longer it takes to compute the password hash. Therefore, the iteration count SHOULD be as large as verification server performance will allow, typically at least 10,000 iterations.", "url": "https://pages.nist.gov/800-63-3/sp800-63b.html"}]}

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.