ScaffoldBench

RunSonnet 5.5repeat 1

Gated-execute on OWASP vs NIST on password hashing

claude-sonnet-5-5__gated-execute__research-password-hashing__r1

Score

0.875

wrong answer
design ran as designed
Cost
$0.112
+ $0.093 judging
Latency
30.2s
setup 0.0s, grading 5.2s
Turns
9
Tool calls
8
Subagents
0
Tokens in / out
49k / 4.5k
33,088 read from cache

Did the design run?

every check held
  • Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
  • No API failures: heldharness.apiNo model call failed at the API.
  • Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
  • No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
  • Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
  • No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
  • Builtin tools: heldharness.builtin-toolsA check recorded with the run.
  • Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
  • No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
  • Judge answered: heldharness.judgeAt least one grading call returned a usable verdict.
  • Every state change was reviewed: heldgated-execute.reviewedEvery call that changed the world had been allowed by a review before it ran, counted from the world’s own journal.
  • Every decision was read: heldgated-execute.decisionsEvery review that ran to its end gave its decision through the schema; an unreadable one is a denial.
  • Each call was shown the right tools: heldgated-execute.toolsThe executor was shown the task’s full toolset and each reviewer the read-only tools.

Phases

Execute: 0.0s to 30.2s, finished, 8 tool calls, $0.112

Drawn from the phase records: timing and cost per call, no tool calls or messages.

PhaseDurationCostTurnsToolsModel
Execute30.2s$0.11298claude-sonnet-5-5

The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.

Runs you can replay

Record this cell yourself: npm run bench -- run --strategies gated-execute --tasks research-password-hashing --models claude-sonnet-5-5 --repeats 1

Score breakdown

  • passStates that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism.
    1.00weight 0.19

    met, backed by verified evidence E3. “Use Argon2id as the default, with at least 19 MiB of memory, 2 iterations and 1 degree of parallelism.”

  • passStates OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required.
    1.00weight 0.13

    met, backed by verified evidence E2, E5. “Use scrypt if Argon2id isn't available.”

  • passStates that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used.
    1.00weight 0.19

    met, backed by verified evidence E6, E7. “passwords "SHALL be salted and hashed using a suitable password hashing scheme"”

  • failStates that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording.
    0.00weight 0.13

    Not stated.

  • passExplains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply.
    1.00weight 0.25

    met, backed by verified evidence E2, E4. “OWASP's FIPS-specific advice is PBKDF2-HMAC-SHA-256 at 600,000 or more iterations. PBKDF2 comes from SP 800-132, and OWASP says it has FIPS-validated implementations.”

  • passStates at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash.
    1.00weight 0.13

    met, backed by verified evidence E1, E6, E8. “Use a unique salt of at least 32 bits.”

Penalties

No penalties were applied.

Scorer metadata

corpus
{"version":2,"snapshot":"2026-10-01","hash":"e12dc326c69a32d91f85f6355dc63e2a7d672b605685a1ffa1582a42bb1c561e","documents":5}
opened documents
["owasp-password-storage","nist-sp800-63b-4"]
opened by role
{"primary":["owasp-password-storage","nist-sp800-63b-4"]}
displayed share
{"owasp-password-storage":0.6794,"nist-sp800-63b-4":0.0628}
cited count
2
verified count
2
hallucinated count
0
citation integrity
1
hallucinated urls
[]
evidence block
true
evidence block repaired
false
evidence count
11
verified quote count
11
verified quote rate
1
evidence
[{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP lists Argon2id, bcrypt and PBKDF2 as acceptable slow hashing algorithms and requires a unique salt","quote":"Passwords should never be stored in plain text. Instead, they must be protected using strong, slow hashing algorithms such as Argon2id, bcrypt, or PBKDF2.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP's FIPS-specific recommendation is PBKDF2 with 600,000+ iterations and HMAC-SHA-256","quote":"If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP's default recommendation is Argon2id with minimum parameters","quote":"Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP says PBKDF2 should be preferred when NIST permission and FIPS validated implementations are needed","quote":"Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP PBKDF2 iteration counts per internal hash","quote":"PBKDF2-HMAC-SHA256: 600,000 iterations (recommended)","verified":true,"document":"owasp-password-storage"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST 800-63B-4 requires salted and hashed password storage with a suitable password hashing scheme","quote":"Passwords SHALL be salted and hashed using a suitable password hashing scheme.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST does not name an algorithm but points to SP 800-132 or updated NIST guidance for approved schemes, and says cost should be high and increase over time","quote":"An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST salt length requirement and storage of salt and hash","quote":"The salt SHALL be at least 32 bits in length and chosen to minimize salt value collisions among stored hashes","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST recommends an additional keyed hash with a secret key stored separately (analogous to a pepper), ideally in an HSM/TEE","quote":"In addition, verifiers SHOULD perform an additional iteration of a keyed hashing or encryption operation using a secret key known only to the verifier.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST requires FIPS 140 Level 1 validated cryptography for federal verifiers at AAL2 (and AAL1)","quote":"Cryptography used by verifiers operated by or on behalf of federal agencies at AAL2 SHALL be validated to meet the requirements of [FIPS140] Level 1 unless otherwise specified.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"FIPS 140 requirements are satisfied via CMVP-validated cryptographic implementations","quote":"Throughout this document, [FIPS140] requirements are satisfied by security technologies, products, and services that utilize implementations of cryptography validated by the Cryptography Module Validation Program [CMVP].","verified":true,"document":"nist-sp800-63b-4"}]
judge unverified quotes
0
judge verdicts
[{"owasp-argon2id":{"status":"met","support":["E3"]},"owasp-conditions":{"status":"met","support":["E2","E5"]},"nist-current":{"status":"met","support":["E6","E7"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E2","E4"]},"common-ground":{"status":"met","support":["E1","E6","E8"]}},{"owasp-argon2id":{"status":"met","support":["E3"]},"owasp-conditions":{"status":"met","support":["E2","E5"]},"nist-current":{"status":"met","support":["E6","E7"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E2","E4","E7"]},"common-ground":{"status":"met","support":["E1","E6","E8"]}},{"owasp-argon2id":{"status":"met","support":["E3"]},"owasp-conditions":{"status":"met","support":["E2","E5"]},"nist-current":{"status":"met","support":["E6","E7"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E2","E4"]},"common-ground":{"status":"met","support":["E1","E6","E8"]}}]
judge failures
0
grounded weight share
0.875 (88%)

Final answer

Short answer: If FIPS-validated cryptography is a hard requirement, use PBKDF2 with HMAC-SHA-256 at 600,000 or more iterations, with a unique salt. If it isn't a hard requirement, OWASP prefers Argon2id. NIST SP 800-63B-4 doesn't name any algorithm, so OWASP's PBKDF2 advice doesn't contradict it.

What OWASP says

  • OWASP's Password Storage Cheat Sheet says passwords "must be protected using strong, slow hashing algorithms such as Argon2id, bcrypt, or PBKDF2". It also says a unique salt must be added to each password. (https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html)
  • Its ordered recommendation is:

    • Use Argon2id as the default, with at least 19 MiB of memory, 2 iterations and 1 degree of parallelism.
    • Use scrypt if Argon2id isn't available.
    • Use bcrypt only for legacy systems, with a work factor of 10 or more and a 72-byte password limit.
    • For FIPS, "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256."
  • In its PBKDF2 section it says PBKDF2 "should be the preferred algorithm" when NIST permission and FIPS-140 validated implementations are required.
  • It lists these iteration counts:

    • PBKDF2-HMAC-SHA256: 600,000 (recommended).
    • PBKDF2-HMAC-SHA512: 220,000.
    • PBKDF2-HMAC-SHA1: 1,400,000, for legacy use only. It says not to choose SHA-1 for new systems.
  • It recommends a pepper for defense in depth. A pepper is a secret shared across all passwords and stored outside the database.
  • It recommends storing the algorithm and work factor with each hash, for example in the PHC string format, so you can upgrade later.

What NIST SP 800-63B-4 says

I read the NIST-hosted copy of SP 800-63B-4, in section 3.1.1.2 on password verifiers (https://pages.nist.gov/800-63-4/sp800-63b.html).

  • Verifiers "SHALL store passwords in a form that is resistant to offline attacks", and passwords "SHALL be salted and hashed using a suitable password hashing scheme".
  • The salt must be at least 32 bits and chosen to avoid collisions between accounts. The salt and the hash must both be stored.
  • The cost factor "SHOULD be as high as practical without negatively impacting verifier performance" and should rise over time.
  • It says an approved scheme "published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used". SP 800-132 is NIST's password-based key derivation guidance, which is where PBKDF2 comes from. The document names no algorithm itself. A search of the page found no occurrence of "PBKDF".
  • It also says a reference to the scheme and cost factor SHOULD be stored with each password to allow migration.
  • Verifiers SHOULD add a further keyed hash or encryption step using a secret key held only by the verifier. This is the same idea as OWASP's pepper. The key must come from an approved random bit generator and be stored separately from the hashes, ideally in an HSM or TEE.
  • On FIPS 140, the document says that FIPS 140 requirements are met by cryptography validated under the Cryptography Module Validation Program (CMVP). Cryptography used by verifiers operated by or on behalf of federal agencies must be validated at FIPS 140 Level 1 at AAL1 and AAL2. Authenticators at AAL1 need only approved algorithms, not validation.

How to reconcile them

  1. The two sources agree. NIST requires a salted, cost-factored scheme from an approved NIST source and leaves the choice of algorithm open. OWASP's FIPS-specific advice is PBKDF2-HMAC-SHA-256 at 600,000 or more iterations. PBKDF2 comes from SP 800-132, and OWASP says it has FIPS-validated implementations.
  2. OWASP states that PBKDF2 "is permitted by NIST SP 800-63B-4". The NIST text I read only points to SP 800-132 and does not name PBKDF2. The link is therefore indirect, so check SP 800-132 and any newer NIST password-hashing guidance yourself.
  3. Argon2id is OWASP's first choice, but I did not find it in the NIST text. If the service must use FIPS-validated cryptography, use PBKDF2 through a CMVP-validated cryptographic module. A library that merely implements PBKDF2 doesn't satisfy FIPS. If FIPS applies only to part of your system, consider Argon2id for the rest, but that is my suggestion rather than something either source says.
  4. Follow NIST's other requirements as well:

    • Use a unique salt of at least 32 bits.
    • Tune the iteration count to your hardware and raise it over time.
    • Store the scheme and cost with each hash.
    • Add a secret-key step, an HMAC with a key kept in an HSM or similar. OWASP's pepper is the same idea.
  5. One caveat on FIPS scope: NIST ties the FIPS 140 validation requirement to verifiers run by or for federal agencies. Check with your compliance team whether your service falls under it.
{"evidence": [
{"claim": "OWASP lists Argon2id, bcrypt and PBKDF2 as acceptable slow hashing algorithms and requires a unique salt", "quote": "Passwords should never be stored in plain text. Instead, they must be protected using strong, slow hashing algorithms such as Argon2id, bcrypt, or PBKDF2.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP's FIPS-specific recommendation is PBKDF2 with 600,000+ iterations and HMAC-SHA-256", "quote": "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP's default recommendation is Argon2id with minimum parameters", "quote": "Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP says PBKDF2 should be preferred when NIST permission and FIPS validated implementations are needed", "quote": "Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP PBKDF2 iteration counts per internal hash", "quote": "PBKDF2-HMAC-SHA256: 600,000 iterations (recommended)", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "NIST 800-63B-4 requires salted and hashed password storage with a suitable password hashing scheme", "quote": "Passwords SHALL be salted and hashed using a suitable password hashing scheme.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST does not name an algorithm but points to SP 800-132 or updated NIST guidance for approved schemes, and says cost should be high and increase over time", "quote": "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST salt length requirement and storage of salt and hash", "quote": "The salt SHALL be at least 32 bits in length and chosen to minimize salt value collisions among stored hashes", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST recommends an additional keyed hash with a secret key stored separately (analogous to a pepper), ideally in an HSM/TEE", "quote": "In addition, verifiers SHOULD perform an additional iteration of a keyed hashing or encryption operation using a secret key known only to the verifier.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST requires FIPS 140 Level 1 validated cryptography for federal verifiers at AAL2 (and AAL1)", "quote": "Cryptography used by verifiers operated by or on behalf of federal agencies at AAL2 SHALL be validated to meet the requirements of [FIPS140] Level 1 unless otherwise specified.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "FIPS 140 requirements are satisfied via CMVP-validated cryptographic implementations", "quote": "Throughout this document, [FIPS140] requirements are satisfied by security technologies, products, and services that utilize implementations of cryptography validated by the Cryptography Module Validation Program [CMVP].", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"}
]}

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.