ScaffoldBench

RunSonnet 5.5repeat 2

Gated-execute on OWASP vs NIST on password hashing

claude-sonnet-5-5__gated-execute__research-password-hashing__r2

Score

0.875

wrong answer
design ran as designed
Cost
$0.090
+ $0.073 judging
Latency
24.1s
setup 0.0s, grading 4.7s
Turns
6
Tool calls
5
Subagents
0
Tokens in / out
38k / 3.5k
23,985 read from cache

Did the design run?

every check held
  • Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
  • No API failures: heldharness.apiNo model call failed at the API.
  • Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
  • No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
  • Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
  • No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
  • Builtin tools: heldharness.builtin-toolsA check recorded with the run.
  • Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
  • No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
  • Judge answered: heldharness.judgeAt least one grading call returned a usable verdict.
  • Every state change was reviewed: heldgated-execute.reviewedEvery call that changed the world had been allowed by a review before it ran, counted from the world’s own journal.
  • Every decision was read: heldgated-execute.decisionsEvery review that ran to its end gave its decision through the schema; an unreadable one is a denial.
  • Each call was shown the right tools: heldgated-execute.toolsThe executor was shown the task’s full toolset and each reviewer the read-only tools.

Phases

Execute: 0.0s to 24.1s, finished, 5 tool calls, $0.090

Drawn from the phase records: timing and cost per call, no tool calls or messages.

PhaseDurationCostTurnsToolsModel
Execute24.1s$0.09065claude-sonnet-5-5

The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.

Runs you can replay

Record this cell yourself: npm run bench -- run --strategies gated-execute --tasks research-password-hashing --models claude-sonnet-5-5 --repeats 1

Score breakdown

  • passStates that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism.
    1.00weight 0.19

    met, backed by verified evidence E1. “Argon2id, at minimum 19 MiB of memory, 2 iterations and 1 degree of parallelism.”

  • passStates OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required.
    1.00weight 0.13

    met, backed by verified evidence E2. “scrypt if Argon2id is unavailable.”

  • passStates that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used.
    1.00weight 0.19

    met, backed by verified evidence E6, E7, E8. “It says an approved scheme from the latest revision of SP 800-132, or updated NIST guidance, SHOULD be used.”

  • failStates that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording.
    0.00weight 0.13

    Not stated.

  • passExplains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply.
    1.00weight 0.25

    met, backed by verified evidence E2, E3. “OWASP itself switches to PBKDF2 when FIPS is required.”

  • passStates at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash.
    1.00weight 0.13

    met, backed by verified evidence E4, E6, E8, E9. “Use a unique random salt of at least 32 bits per password, and store the salt, the hash and the parameters.”

Penalties

No penalties were applied.

Scorer metadata

corpus
{"version":2,"snapshot":"2026-10-01","hash":"e12dc326c69a32d91f85f6355dc63e2a7d672b605685a1ffa1582a42bb1c561e","documents":5}
opened documents
["owasp-password-storage","nist-sp800-63b-4"]
opened by role
{"primary":["owasp-password-storage","nist-sp800-63b-4"]}
displayed share
{"owasp-password-storage":0.8205,"nist-sp800-63b-4":0.0618}
cited count
2
verified count
2
hallucinated count
0
citation integrity
1
hallucinated urls
[]
evidence block
true
evidence block repaired
false
evidence count
10
verified quote count
10
verified quote rate
1
evidence
[{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP's general first choice is Argon2id with a minimum configuration of 19 MiB memory, 2 iterations and parallelism 1.","quote":"Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP says to use PBKDF2 with at least 600,000 iterations and HMAC-SHA-256 when FIPS-140 compliance is required.","quote":"If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP states PBKDF2 is permitted by NIST SP 800-63B-4, has FIPS-140 validated implementations, and should be preferred when those are required.","quote":"Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP lists 600,000 iterations as the recommended count for PBKDF2-HMAC-SHA256.","quote":"PBKDF2-HMAC-SHA256: 600,000 iterations (recommended)","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP says a pepper should be stored separately from the password database, in a secrets vault or HSM.","quote":"The pepper should be stored separately from the password database.","verified":true,"document":"owasp-password-storage"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST SP 800-63B-4 requires passwords to be salted and hashed with a suitable password hashing scheme.","quote":"Passwords SHALL be salted and hashed using a suitable password hashing scheme.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST says an approved scheme from the latest SP 800-132 (or updated NIST guidance) should be used, without naming an algorithm in 800-63B-4; the cost factor should be as high as practical.","quote":"An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST says the cost factor should be as high as practical and increased over time.","quote":"The chosen cost factor SHOULD be as high as practical without negatively impacting verifier performance.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST requires a salt of at least 32 bits, and requires both the salt and the hash to be stored.","quote":"The salt SHALL be at least 32 bits in length and chosen to minimize salt value collisions among stored hashes","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST recommends an additional keyed hash or encryption iteration with a secret key stored separately, ideally in an HSM or TEE (analogous to a pepper).","quote":"In addition, verifiers SHOULD perform an additional iteration of a keyed hashing or encryption operation using a secret key known only to the verifier.","verified":true,"document":"nist-sp800-63b-4"}]
judge unverified quotes
0
judge verdicts
[{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2"]},"nist-current":{"status":"met","support":["E6","E7","E8"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E4","E6","E8","E9"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2"]},"nist-current":{"status":"met","support":["E6","E7","E8"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E6","E8","E9"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2"]},"nist-current":{"status":"met","support":["E6","E7","E8"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E6","E9","E8"]}}]
judge failures
0
grounded weight share
0.875 (88%)

Final answer

The OWASP cheat sheet recommends Argon2id in general and PBKDF2 when FIPS-140 compliance is required. NIST SP 800-63B-4 requires salted hashing with an approved scheme but names no algorithm. For a FIPS-bound service, that means PBKDF2.

What OWASP says (https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html)

  • The default order is:

    • Argon2id, at minimum 19 MiB of memory, 2 iterations and 1 degree of parallelism.
    • scrypt if Argon2id is unavailable.
    • bcrypt only for legacy systems.
  • For FIPS, it says to use PBKDF2 with a work factor of 600,000 or more and HMAC-SHA-256 as the internal hash.
  • The detailed PBKDF2 section lists these iteration counts:

    • PBKDF2-HMAC-SHA256: 600,000, which it marks as recommended.
    • PBKDF2-HMAC-SHA512: 220,000.
    • PBKDF2-HMAC-SHA1: 1,400,000, for legacy use only.
  • It also suggests an optional pepper for defense in depth. The pepper should be stored separately from the database, for example in a vault or HSM.
  • It says PBKDF2 "is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations."

What NIST SP 800-63B-4 says (https://pages.nist.gov/800-63-4/sp800-63b.html)

  • Section 3.1.1.2 requires that "Passwords SHALL be salted and hashed using a suitable password hashing scheme."
  • The salt must be at least 32 bits and chosen to minimize collisions. Both the salt and the hash must be stored.
  • The cost factor "SHOULD be as high as practical without negatively impacting verifier performance" and should be raised over time.
  • It says an approved scheme from the latest revision of SP 800-132, or updated NIST guidance, SHOULD be used. It does not name PBKDF2, Argon2id or any other algorithm itself.
  • It says verifiers SHOULD apply an extra keyed-hash or encryption step using a secret key. That key must come from an approved random bit generator and be stored separately from the hashes, ideally in an HSM or TEE. This works like OWASP's pepper.
  • It also says the scheme reference and cost factor SHOULD be stored per password so you can migrate later.
  • I read only the password-storage passage of NIST's document, not the whole thing. I did not open SP 800-132 or any FIPS 140 validation material, so the point that PBKDF2 is that document's approved scheme comes from OWASP's wording plus my own background knowledge.

One difference between the sources. OWASP says PBKDF2 "is permitted by NIST SP 800-63B-4." In the passage I read, NIST only points to SP 800-132 and does not name PBKDF2. The two are compatible, but the link to PBKDF2 goes through SP 800-132. It is not stated in 800-63B-4 itself.

How to reconcile them

  1. OWASP's general preference for Argon2id and NIST's FIPS-oriented requirements do not conflict. OWASP itself switches to PBKDF2 when FIPS is required.
  2. Use PBKDF2-HMAC-SHA256 with at least 600,000 iterations. Raise the count as far as login latency allows, since NIST says the cost should be as high as practical.
  3. Use a unique random salt of at least 32 bits per password, and store the salt, the hash and the parameters.
  4. Add a secret key step (a pepper or keyed HMAC), stored separately and ideally in an HSM. Both OWASP and NIST support this. If your approved cryptographic module requires it, use a FIPS-validated implementation for the HMAC too.
  5. Confirm that your cryptographic library or module is FIPS 140 validated and that your auditor accepts the exact PBKDF2 configuration. OWASP says validated implementations exist, but I did not verify any particular module.
  6. Keep the parameters versioned so you can move to an NIST-approved successor later.
{"evidence": [
{"claim": "OWASP's general first choice is Argon2id with a minimum configuration of 19 MiB memory, 2 iterations and parallelism 1.", "quote": "Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP says to use PBKDF2 with at least 600,000 iterations and HMAC-SHA-256 when FIPS-140 compliance is required.", "quote": "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP states PBKDF2 is permitted by NIST SP 800-63B-4, has FIPS-140 validated implementations, and should be preferred when those are required.", "quote": "Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP lists 600,000 iterations as the recommended count for PBKDF2-HMAC-SHA256.", "quote": "PBKDF2-HMAC-SHA256: 600,000 iterations (recommended)", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP says a pepper should be stored separately from the password database, in a secrets vault or HSM.", "quote": "The pepper should be stored separately from the password database.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "NIST SP 800-63B-4 requires passwords to be salted and hashed with a suitable password hashing scheme.", "quote": "Passwords SHALL be salted and hashed using a suitable password hashing scheme.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST says an approved scheme from the latest SP 800-132 (or updated NIST guidance) should be used, without naming an algorithm in 800-63B-4; the cost factor should be as high as practical.", "quote": "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST says the cost factor should be as high as practical and increased over time.", "quote": "The chosen cost factor SHOULD be as high as practical without negatively impacting verifier performance.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST requires a salt of at least 32 bits, and requires both the salt and the hash to be stored.", "quote": "The salt SHALL be at least 32 bits in length and chosen to minimize salt value collisions among stored hashes", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST recommends an additional keyed hash or encryption iteration with a secret key stored separately, ideally in an HSM or TEE (analogous to a pepper).", "quote": "In addition, verifiers SHOULD perform an additional iteration of a keyed hashing or encryption operation using a secret key known only to the verifier.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"}
]}

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.