ScaffoldBench

RunOpus 5.5repeat 2

Gated-execute on OWASP vs NIST on password hashing

claude-opus-5-5__gated-execute__research-password-hashing__r2

Score

1.000

passed
design ran as designed
Cost
$0.284
+ $0.102 judging
Latency
56.4s
setup 0.0s, grading 5.5s
Turns
11
Tool calls
10
Subagents
0
Tokens in / out
101k / 6.1k
81,568 read from cache

Did the design run?

every check held
  • Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
  • No API failures: heldharness.apiNo model call failed at the API.
  • Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
  • No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
  • Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
  • No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
  • Builtin tools: heldharness.builtin-toolsA check recorded with the run.
  • Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
  • No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
  • Judge answered: heldharness.judgeAt least one grading call returned a usable verdict.
  • Every state change was reviewed: heldgated-execute.reviewedEvery call that changed the world had been allowed by a review before it ran, counted from the world’s own journal.
  • Every decision was read: heldgated-execute.decisionsEvery review that ran to its end gave its decision through the schema; an unreadable one is a denial.
  • Each call was shown the right tools: heldgated-execute.toolsThe executor was shown the task’s full toolset and each reviewer the read-only tools.

Phases

Execute: 0.0s to 56.4s, finished, 10 tool calls, $0.284

Drawn from the phase records: timing and cost per call, no tool calls or messages.

PhaseDurationCostTurnsToolsModel
Execute56.4s$0.2841110claude-opus-5-5

The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.

Runs you can replay

Record this cell yourself: npm run bench -- run --strategies gated-execute --tasks research-password-hashing --models claude-opus-5-5 --repeats 1

Score breakdown

  • passStates that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism.
    1.00weight 0.19

    met, backed by verified evidence E1. “**Argon2id** first, at a minimum of 19 MiB memory, 2 iterations and parallelism 1.”

  • passStates OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required.
    1.00weight 0.13

    met, backed by verified evidence E2, E3, E4. “**scrypt** if Argon2id isn't available.”

  • passStates that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used.
    1.00weight 0.19

    met, backed by verified evidence E9, E10, E11. “Use an approved scheme from the latest revision of SP 800-132 or later NIST password-hashing guidance.”

  • passStates that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording.
    1.00weight 0.13

    met, backed by verified evidence E16, E17. “it named "PBKDF2 [SP 800-132] and Balloon" as examples, said a memory-hard function SHOULD be used, and suggested "typically at least 10,000 iterations"”

  • passExplains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply.
    1.00weight 0.25

    met, backed by verified evidence E4, E5. “Don't pick Argon2id, scrypt or bcrypt for this service. OWASP's own FIPS rule points to PBKDF2 instead.”

  • passStates at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash.
    1.00weight 0.13

    met, backed by verified evidence E12, E11. “It also requires a unique salt for each password”

Penalties

No penalties were applied.

Scorer metadata

corpus
{"version":2,"snapshot":"2026-10-01","hash":"e12dc326c69a32d91f85f6355dc63e2a7d672b605685a1ffa1582a42bb1c561e","documents":5}
opened documents
["owasp-password-storage","nist-sp800-63b-4","nist-sp800-63b-3"]
opened by role
{"primary":["owasp-password-storage","nist-sp800-63b-4"],"superseded":["nist-sp800-63b-3"]}
displayed share
{"owasp-password-storage":0.8306,"nist-sp800-63b-4":0.0628,"nist-sp800-63b-3":0.0456,"rfc9106":0.009,"owasp-authentication":0.007}
cited count
3
verified count
3
hallucinated count
0
citation integrity
1
hallucinated urls
[]
evidence block
true
evidence block repaired
false
evidence count
17
verified quote count
17
verified quote rate
1
evidence
[{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP's general recommendation is Argon2id with minimum parameters 19 MiB, 2 iterations, parallelism 1.","quote":"Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP recommends scrypt when Argon2id is unavailable.","quote":"If Argon2id is not available, use scrypt with a minimum CPU/memory cost parameter of (2^17), a minimum block size of 8 (1024 bytes), and a parallelization parameter of 1.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP limits bcrypt to legacy systems.","quote":"The bcrypt password hashing function should only be used for password storage in legacy systems where Argon2 and scrypt are not available.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"For FIPS-140 compliance OWASP specifies PBKDF2 with HMAC-SHA-256 and at least 600,000 iterations.","quote":"If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP says PBKDF2 should be preferred when NIST 800-63B-4 or FIPS-140 compliance is required.","quote":"Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP calls PBKDF2-HMAC-SHA1 legacy only.","quote":"PBKDF2-HMAC-SHA1: 1,400,000 iterations — legacy only, do not select for new systems.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP's PBKDF2 iteration figures date from December 2022 GPU testing.","quote":"These configuration settings are equivalent in the defense they provide. (Number as of december 2022, based on testing of RTX 4000 GPUs)","verified":true,"document":"owasp-password-storage"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"SP 800-63B-4 is the current revision and supersedes the earlier SP 800-63B.","quote":"This publication supersedes NIST Special Publication (SP) 800-63B.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST 800-63B-4 requires passwords to be salted and hashed with a suitable password hashing scheme.","quote":"Passwords SHALL be salted and hashed using a suitable password hashing scheme.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST 800-63B-4 says an approved scheme from SP 800-132 SHOULD be used (a recommendation, not a SHALL).","quote":"An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST 800-63B-4 sets no fixed iteration count; the cost should be as high as practical and increased over time.","quote":"The chosen cost factor SHOULD be as high as practical without negatively impacting verifier performance.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST 800-63B-4 requires a salt of at least 32 bits.","quote":"The salt SHALL be at least 32 bits in length and chosen to minimize salt value collisions among stored hashes (i.e., to prevent multiple subscriber accounts from having the same hashed password).","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST 800-63B-4 recommends storing the scheme and cost factor so passwords can be migrated.","quote":"A reference to the password hashing scheme used, including the cost factor, SHOULD be stored for each password to allow migration to new algorithms and work factors.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST 800-63B-4 recommends an extra keyed hash or encryption step with a verifier-only secret key.","quote":"In addition, verifiers SHOULD perform an additional iteration of a keyed hashing or encryption operation using a secret key known only to the verifier.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST 800-63B-4 requires the secret key to be stored separately from the hashes.","quote":"The secret key value SHALL be stored separately from the hashed passwords.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-3/sp800-63b.html","claim":"The previous revision (800-63B rev 3) named PBKDF2 (SP 800-132) and Balloon as examples, preferred memory-hard functions, and suggested at least 10,000 PBKDF2 iterations.","quote":"Examples of suitable key derivation functions include Password-based Key Derivation Function 2 (PBKDF2) [SP 800-132] and Balloon [BALLOON].","verified":true,"document":"nist-sp800-63b-3"},{"url":"https://pages.nist.gov/800-63-3/sp800-63b.html","claim":"The previous revision's PBKDF2 iteration guidance was typically at least 10,000.","quote":"Therefore, the iteration count SHOULD be as large as verification server performance will allow, typically at least 10,000 iterations.","verified":true,"document":"nist-sp800-63b-3"}]
judge unverified quotes
0
judge verdicts
[{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2","E3","E4"]},"nist-current":{"status":"met","support":["E9","E10","E11"]},"nist-revision":{"status":"met","support":["E16","E17"]},"reconciliation":{"status":"met","support":["E4","E5"]},"common-ground":{"status":"met","support":["E12","E11"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2","E3","E4"]},"nist-current":{"status":"met","support":["E9","E10","E11"]},"nist-revision":{"status":"met","support":["E16","E17"]},"reconciliation":{"status":"met","support":["E4","E5"]},"common-ground":{"status":"met","support":["E9","E12"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2","E3","E4"]},"nist-current":{"status":"met","support":["E9","E10","E11"]},"nist-revision":{"status":"met","support":["E16","E17"]},"reconciliation":{"status":"met","support":["E4","E5"]},"common-ground":{"status":"met","support":["E9","E12","E11"]}}]
judge failures
0
grounded weight share
1

Final answer

Recommendation: use PBKDF2 with HMAC-SHA-256 and at least 600,000 iterations, run inside a FIPS 140-validated crypto module. Add a unique salt for each password and a secret key ("pepper") kept in an HSM. That one design satisfies OWASP's FIPS-specific advice and the storage requirements in NIST SP 800-63B-4.

What OWASP recommends

The OWASP Password Storage Cheat Sheet (https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html) ranks the algorithms in this order:

  • Argon2id first, at a minimum of 19 MiB memory, 2 iterations and parallelism 1.
  • scrypt if Argon2id isn't available.
  • bcrypt only for legacy systems, with a work factor of at least 10 and a 72-byte password limit.

It gives a separate rule for FIPS: "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256." It explains that PBKDF2 "is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations," so it should be the preferred choice when those are required. It also gives equivalent iteration counts for other hash functions:

  • 220,000 for PBKDF2-HMAC-SHA512
  • 1,400,000 for PBKDF2-HMAC-SHA1, which it calls "legacy only, do not select for new systems"

The cheat sheet dates these numbers to December 2022 testing on RTX 4000 GPUs. It also requires a unique salt for each password and says to consider a pepper, though it says a pepper alone adds no security.

What NIST SP 800-63B-4 requires

This is the current revision. It states that it "supersedes NIST Special Publication (SP) 800-63B" (https://pages.nist.gov/800-63-4/sp800-63b.html). It doesn't name an algorithm. Its rules are:

  • Required (SHALL):

    • Passwords must be "salted and hashed using a suitable password hashing scheme."
    • The salt must be at least 32 bits.
    • The salt and the hash must both be stored.
    • If a secret key is used, it must come from an approved random bit generator and be stored separately from the hashes.
  • Recommended (SHOULD):

    • Use an approved scheme from the latest revision of SP 800-132 or later NIST password-hashing guidance.
    • Set the cost factor as high as practical and raise it over time.
    • Store which scheme and cost factor each password used, so you can migrate later.
    • Add a keyed hash or encryption step with a secret key, ideally held in an HSM or a trusted execution environment.

The new revision doesn't give an iteration count. The previous revision did: it named "PBKDF2 [SP 800-132] and Balloon" as examples, said a memory-hard function SHOULD be used, and suggested "typically at least 10,000 iterations" (https://pages.nist.gov/800-63-3/sp800-63b.html). Revision 4 dropped all three points.

Where the two sources differ

  • Preferred algorithm. OWASP's general choice, Argon2id, isn't the scheme NIST points to. NIST points to SP 800-132, which the previous revision identifies as the PBKDF2 standard. (I didn't open SP 800-132 itself.) NIST's wording here is a SHOULD rather than an absolute ban. In practice, FIPS 140 validation is what rules out other algorithms; that reasoning comes from OWASP, and NIST doesn't say it.
  • Iteration count. OWASP's 600,000 is a concrete benchmark. NIST only says "as high as practical."
  • Pepper. NIST recommends it (SHOULD). OWASP only says to "consider" it.

How to reconcile them

  1. Use PBKDF2-HMAC-SHA-256 from a FIPS 140-validated module, at 600,000 iterations or more. Tune it upward over time; that covers both OWASP's minimum and NIST's "as high as practical."
  2. Give every password a unique random salt of at least 32 bits (NIST's minimum). I'd suggest more; that's my own suggestion, not from either source.
  3. Store the salt, the hash, and the scheme and cost parameters with each password.
  4. Add an HMAC step keyed with a secret pepper. Generate the key with an approved random bit generator and keep it separately, ideally in an HSM.
  5. Don't use PBKDF2-HMAC-SHA1 for new systems.
  6. Don't pick Argon2id, scrypt or bcrypt for this service. OWASP's own FIPS rule points to PBKDF2 instead.
{"evidence": [
{"claim": "OWASP's general recommendation is Argon2id with minimum parameters 19 MiB, 2 iterations, parallelism 1.", "quote": "Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP recommends scrypt when Argon2id is unavailable.", "quote": "If Argon2id is not available, use scrypt with a minimum CPU/memory cost parameter of (2^17), a minimum block size of 8 (1024 bytes), and a parallelization parameter of 1.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP limits bcrypt to legacy systems.", "quote": "The bcrypt password hashing function should only be used for password storage in legacy systems where Argon2 and scrypt are not available.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "For FIPS-140 compliance OWASP specifies PBKDF2 with HMAC-SHA-256 and at least 600,000 iterations.", "quote": "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP says PBKDF2 should be preferred when NIST 800-63B-4 or FIPS-140 compliance is required.", "quote": "Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP calls PBKDF2-HMAC-SHA1 legacy only.", "quote": "PBKDF2-HMAC-SHA1: 1,400,000 iterations — legacy only, do not select for new systems.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP's PBKDF2 iteration figures date from December 2022 GPU testing.", "quote": "These configuration settings are equivalent in the defense they provide. (Number as of december 2022, based on testing of RTX 4000 GPUs)", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "SP 800-63B-4 is the current revision and supersedes the earlier SP 800-63B.", "quote": "This publication supersedes NIST Special Publication (SP) 800-63B.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST 800-63B-4 requires passwords to be salted and hashed with a suitable password hashing scheme.", "quote": "Passwords SHALL be salted and hashed using a suitable password hashing scheme.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST 800-63B-4 says an approved scheme from SP 800-132 SHOULD be used (a recommendation, not a SHALL).", "quote": "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST 800-63B-4 sets no fixed iteration count; the cost should be as high as practical and increased over time.", "quote": "The chosen cost factor SHOULD be as high as practical without negatively impacting verifier performance.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST 800-63B-4 requires a salt of at least 32 bits.", "quote": "The salt SHALL be at least 32 bits in length and chosen to minimize salt value collisions among stored hashes (i.e., to prevent multiple subscriber accounts from having the same hashed password).", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST 800-63B-4 recommends storing the scheme and cost factor so passwords can be migrated.", "quote": "A reference to the password hashing scheme used, including the cost factor, SHOULD be stored for each password to allow migration to new algorithms and work factors.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST 800-63B-4 recommends an extra keyed hash or encryption step with a verifier-only secret key.", "quote": "In addition, verifiers SHOULD perform an additional iteration of a keyed hashing or encryption operation using a secret key known only to the verifier.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST 800-63B-4 requires the secret key to be stored separately from the hashes.", "quote": "The secret key value SHALL be stored separately from the hashed passwords.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "The previous revision (800-63B rev 3) named PBKDF2 (SP 800-132) and Balloon as examples, preferred memory-hard functions, and suggested at least 10,000 PBKDF2 iterations.", "quote": "Examples of suitable key derivation functions include Password-based Key Derivation Function 2 (PBKDF2) [SP 800-132] and Balloon [BALLOON].", "url": "https://pages.nist.gov/800-63-3/sp800-63b.html"},
{"claim": "The previous revision's PBKDF2 iteration guidance was typically at least 10,000.", "quote": "Therefore, the iteration count SHOULD be as large as verification server performance will allow, typically at least 10,000 iterations.", "url": "https://pages.nist.gov/800-63-3/sp800-63b.html"}
]}

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.