ScaffoldBench

RunHaiku 4.5repeat 2

Gated-execute on OWASP vs NIST on password hashing

claude-haiku-4-5-20251001__gated-execute__research-password-hashing__r2

Score

0.641

wrong answer
design ran as designed
Cost
$0.178
+ $0.083 judging
Latency
1m 26s
setup 0.0s, grading 4.6s
Turns
34
Tool calls
33
Subagents
0
Tokens in / out
643k / 7.3k
598,456 read from cache

Did the design run?

every check held
  • Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
  • No API failures: heldharness.apiNo model call failed at the API.
  • Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
  • No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
  • Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
  • No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
  • Builtin tools: heldharness.builtin-toolsA check recorded with the run.
  • Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
  • No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
  • Judge answered: heldharness.judgeAt least one grading call returned a usable verdict.
  • Every state change was reviewed: heldgated-execute.reviewedEvery call that changed the world had been allowed by a review before it ran, counted from the world’s own journal.
  • Every decision was read: heldgated-execute.decisionsEvery review that ran to its end gave its decision through the schema; an unreadable one is a denial.
  • Each call was shown the right tools: heldgated-execute.toolsThe executor was shown the task’s full toolset and each reviewer the read-only tools.

Phases

Execute: 0.0s to 1m 26s, finished, 33 tool calls, $0.178

Drawn from the phase records: timing and cost per call, no tool calls or messages.

PhaseDurationCostTurnsToolsModel
Execute1m 26s$0.1783433claude-haiku-4-5-20251001

The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.

Runs you can replay

Record this cell yourself: npm run bench -- run --strategies gated-execute --tasks research-password-hashing --models claude-haiku-4-5-20251001 --repeats 1

Score breakdown

  • passStates that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism.
    1.00weight 0.19

    met, backed by verified evidence E1. “**Argon2id** (first choice) with minimum configuration of 19 MiB of memory, 2 iterations, and 1 degree of parallelism”

  • passStates OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required.
    1.00weight 0.13

    met, backed by verified evidence E2. “OWASP recommends a different approach: **PBKDF2 with a work factor of 600,000 or more iterations and HMAC-SHA-256 as the internal hash function**.”

  • passStates that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used.
    1.00weight 0.19

    met, backed by verified evidence E4, E5. “"An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used."”

  • failStates that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording.
    0.00weight 0.13

    Not stated.

  • partialExplains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply.
    0.50weight 0.25

    partial, backed by verified evidence E2, E3. “Use PBKDF2 with HMAC-SHA-256 and 600,000+ iterations** to satisfy both NIST SP 800-63B-4 (as an approved scheme per SP 800-132) and FIPS 140 requirements”

  • partialStates at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash.
    0.13weight 0.13

    partial, but no evidence entry supports it. “OWASP's emphasis on storing algorithm and cost factor metadata supports this strategy”

Penalties

No penalties were applied.

Scorer metadata

corpus
{"version":2,"snapshot":"2026-10-01","hash":"e12dc326c69a32d91f85f6355dc63e2a7d672b605685a1ffa1582a42bb1c561e","documents":5}
opened documents
["owasp-password-storage","nist-sp800-63b-4"]
opened by role
{"primary":["owasp-password-storage","nist-sp800-63b-4"]}
displayed share
{"owasp-password-storage":1,"owasp-authentication":0.0682,"rfc9106":0.0818,"nist-sp800-63b-4":0.151,"nist-sp800-63b-3":0.0134}
cited count
2
verified count
2
hallucinated count
0
citation integrity
1
hallucinated urls
[]
evidence block
true
evidence block repaired
false
evidence count
7
verified quote count
7
verified quote rate
1
evidence
[{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP recommends Argon2id as the primary password hashing algorithm with minimum 19 MiB of memory, 2 iterations, and 1 degree of parallelism","quote":"Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP recommends PBKDF2 for FIPS-140 compliance with 600,000 or more iterations and HMAC-SHA-256","quote":"If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, making it the preferred algorithm when these are required","quote":"Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.","verified":true,"document":"owasp-password-storage"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST SP 800-63B-4 requires that passwords be salted and hashed using a suitable password hashing scheme","quote":"Passwords SHALL be salted and hashed using a suitable password hashing scheme.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST SP 800-63B-4 recommends using an approved password hashing scheme from SP 800-132 or updated NIST guidelines","quote":"An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST requires cryptography used by federal agency verifiers to be validated to FIPS 140 Level 1 requirements","quote":"Cryptography used by verifiers operated by or on behalf of federal agencies at AAL1 SHALL be validated to meet the requirements of [FIPS140] Level 1.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP Password Storage Cheat Sheet recommends storing a reference to the password hashing scheme used, including the cost factor, to allow migration to new algorithms","quote":"Remember that once your password hashing method is selected, it will have to be upgraded in the future, so ensure that upgrading your hashing algorithm is as easy as possible.","verified":true,"document":"owasp-password-storage"}]
judge unverified quotes
0
judge verdicts
[{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2"]},"nist-current":{"status":"met","support":["E4","E5"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"partial","support":["E2","E3"]},"common-ground":{"status":"partial","support":[]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2"]},"nist-current":{"status":"met","support":["E4","E5"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"partial","support":["E2","E3"]},"common-ground":{"status":"partial","support":["E4"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2"]},"nist-current":{"status":"met","support":["E4","E5"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"partial","support":["E2","E3"]},"common-ground":{"status":"partial","support":[]}}]
judge failures
0
grounded weight share
0.75 (75%)

Final answer

Now I have comprehensive information from both sources. Let me compile the findings into a complete answer.

Answer

OWASP Password Storage Guidance

OWASP recommends a tiered approach to password hashing based on availability and compliance requirements. According to the OWASP Password Storage Cheat Sheet, the preferred algorithms for secure password storage are:

  1. Argon2id (first choice) with minimum configuration of 19 MiB of memory, 2 iterations, and 1 degree of parallelism
  2. scrypt (if Argon2id unavailable) with minimum CPU/memory cost parameter of 2^17, block size of 8 (1024 bytes), and parallelization parameter of 1
  3. bcrypt (for legacy systems only) with a work factor of 10 or more

However, when FIPS-140 compliance is required, OWASP recommends a different approach: PBKDF2 with a work factor of 600,000 or more iterations and HMAC-SHA-256 as the internal hash function.

NIST SP 800-63B-4 Requirements

The current revision of NIST SP 800-63B-4 (published 2023) provides more cautious guidance on password hashing. The standard specifies that "Passwords SHALL be salted and hashed using a suitable password hashing scheme." Regarding specific algorithms, NIST 800-63B-4 states: "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used."

Notably, NIST 800-63B-4 does not name specific algorithms like Argon2id, scrypt, or bcrypt. Instead, it defers to SP 800-132 (from 2010), which focuses on PBKDF2 as a password-based key derivation function. The standard requires that at least 32 bits of salt be used, that the salt and hash be stored with each password, and that the hashing scheme reference and cost factor be stored to enable future algorithm migration.

For federal agencies, NIST 800-63B-4 requires that "Cryptography used by verifiers operated by or on behalf of federal agencies" must be validated to meet FIPS 140 Level 1 requirements.

Reconciling the Two Approaches

The apparent tension between OWASP and NIST SP 800-63B-4 can be reconciled by understanding that they address different priorities:

  1. OWASP's Recommendation Hierarchy: OWASP prioritizes modern cryptographic best practices for maximum security. Argon2id is recommended because it specifically won the 2015 Password Hashing Competition and resists both GPU and memory-based attacks. PBKDF2 is acceptable but is ranked lower on the preference ladder.
  2. NIST's Conservative Approach: NIST SP 800-63B-4 uses the word "SHOULD" rather than "SHALL" when recommending approved schemes, indicating flexibility. By referencing SP 800-132 and "updated NIST guidelines," NIST creates a pathway for modern algorithms without formally standardizing them in 800-63B-4 itself.
  3. FIPS 140 Compliance: This is where the two sources genuinely diverge for practical implementation. While modern algorithms like Argon2id are stronger, FIPS 140-2/140-3 validated implementations are currently rare for these algorithms. PBKDF2 has extensive FIPS 140 validated implementations because it's based on HMAC and standard hash functions (like SHA-256) that are already FIPS-approved.
  4. Recommended Reconciliation Strategy: For teams needing FIPS compliance, the practical solution is:

    • Use PBKDF2 with HMAC-SHA-256 and 600,000+ iterations to satisfy both NIST SP 800-63B-4 (as an approved scheme per SP 800-132) and FIPS 140 requirements
    • Implement a migration path to Argon2id or another modern algorithm once FIPS-validated implementations become more widely available
    • OWASP's emphasis on storing algorithm and cost factor metadata supports this strategy, as it enables algorithm upgrades when security needs or compliance rules change

The key insight is that PBKDF2 is not OWASP's first choice for security reasons, but it is the pragmatic choice when federal compliance mandates FIPS-validated cryptography.


{"evidence": [{"claim": "OWASP recommends Argon2id as the primary password hashing algorithm with minimum 19 MiB of memory, 2 iterations, and 1 degree of parallelism", "quote": "Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"}, {"claim": "OWASP recommends PBKDF2 for FIPS-140 compliance with 600,000 or more iterations and HMAC-SHA-256", "quote": "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"}, {"claim": "PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, making it the preferred algorithm when these are required", "quote": "Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"}, {"claim": "NIST SP 800-63B-4 requires that passwords be salted and hashed using a suitable password hashing scheme", "quote": "Passwords SHALL be salted and hashed using a suitable password hashing scheme.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"}, {"claim": "NIST SP 800-63B-4 recommends using an approved password hashing scheme from SP 800-132 or updated NIST guidelines", "quote": "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"}, {"claim": "NIST requires cryptography used by federal agency verifiers to be validated to FIPS 140 Level 1 requirements", "quote": "Cryptography used by verifiers operated by or on behalf of federal agencies at AAL1 SHALL be validated to meet the requirements of [FIPS140] Level 1.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"}, {"claim": "OWASP Password Storage Cheat Sheet recommends storing a reference to the password hashing scheme used, including the cost factor, to allow migration to new algorithms", "quote": "Remember that once your password hashing method is selected, it will have to be upgraded in the future, so ensure that upgrading your hashing algorithm is as easy as possible.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"}]}

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.