ScaffoldBench

RunOpus 5.5repeat 1

Gated-execute on OWASP vs NIST on password hashing

claude-opus-5-5__gated-execute__research-password-hashing__r1

Score

1.000

passed
design ran as designed
Cost
$0.243
+ $0.083 judging
Latency
46.4s
setup 0.0s, grading 5.1s
Turns
8
Tool calls
7
Subagents
0
Tokens in / out
71k / 4.9k
52,850 read from cache

Did the design run?

every check held
  • Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
  • No API failures: heldharness.apiNo model call failed at the API.
  • Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
  • No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
  • Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
  • No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
  • Builtin tools: heldharness.builtin-toolsA check recorded with the run.
  • Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
  • No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
  • Judge answered: heldharness.judgeAt least one grading call returned a usable verdict.
  • Every state change was reviewed: heldgated-execute.reviewedEvery call that changed the world had been allowed by a review before it ran, counted from the world’s own journal.
  • Every decision was read: heldgated-execute.decisionsEvery review that ran to its end gave its decision through the schema; an unreadable one is a denial.
  • Each call was shown the right tools: heldgated-execute.toolsThe executor was shown the task’s full toolset and each reviewer the read-only tools.

Phases

Execute: 0.0s to 46.4s, finished, 7 tool calls, $0.243

Drawn from the phase records: timing and cost per call, no tool calls or messages.

PhaseDurationCostTurnsToolsModel
Execute46.4s$0.24387claude-opus-5-5

The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.

Runs you can replay

Record this cell yourself: npm run bench -- run --strategies gated-execute --tasks research-password-hashing --models claude-opus-5-5 --repeats 1

Score breakdown

  • passStates that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism.
    1.00weight 0.19

    met, backed by verified evidence E1. “Argon2id, at minimum 19 MiB of memory, 2 iterations and parallelism of 1.”

  • passStates OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required.
    1.00weight 0.13

    met, backed by verified evidence E2, E4. “scrypt, if Argon2id isn't available.”

  • passStates that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used.
    1.00weight 0.19

    met, backed by verified evidence E7, E8. “recommends (SHOULD, not SHALL) a scheme approved by SP 800-132”

  • passStates that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording.
    1.00weight 0.13

    met, backed by verified evidence E12. “The older Revision 3 (https://pages.nist.gov/800-63-3/sp800-63b.html) named PBKDF2 and Balloon as examples. It said "A memory-hard function SHOULD be used"”

  • passExplains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply.
    1.00weight 0.25

    met, backed by verified evidence E2, E3. “The sources don't conflict. OWASP's general preference for Argon2id doesn't override NIST, and OWASP itself makes an exception for FIPS.”

  • passStates at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash.
    1.00weight 0.13

    met, backed by verified evidence E10, E9. “Use a unique salt of at least 32 bits for each password, stored with the hash.”

Penalties

No penalties were applied.

Scorer metadata

corpus
{"version":2,"snapshot":"2026-10-01","hash":"e12dc326c69a32d91f85f6355dc63e2a7d672b605685a1ffa1582a42bb1c561e","documents":5}
opened documents
["owasp-password-storage","nist-sp800-63b-4","nist-sp800-63b-3"]
opened by role
{"primary":["owasp-password-storage","nist-sp800-63b-4"],"superseded":["nist-sp800-63b-3"]}
displayed share
{"owasp-password-storage":0.8205,"nist-sp800-63b-3":0.0464,"nist-sp800-63b-4":0.0619,"rfc9106":0.009,"owasp-authentication":0.0139}
cited count
3
verified count
3
hallucinated count
0
citation integrity
1
hallucinated urls
[]
evidence block
true
evidence block repaired
false
evidence count
13
verified quote count
13
verified quote rate
1
evidence
[{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP's summary recommends Argon2id first, with minimum parameters.","quote":"Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP recommends PBKDF2 with HMAC-SHA-256 and at least 600,000 iterations when FIPS-140 compliance is required.","quote":"If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP justifies PBKDF2 for FIPS cases by citing NIST SP 800-63B-4 and FIPS-140-validated implementations.","quote":"Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP limits bcrypt to legacy systems.","quote":"The bcrypt password hashing function should only be used for password storage in legacy systems where Argon2 and scrypt are not available.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP suggests a pepper as defense in depth.","quote":"Consider using a pepper to provide additional defense in depth (though alone, it provides no additional secure characteristics).","verified":true,"document":"owasp-password-storage"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"SP 800-63B-4 supersedes the earlier SP 800-63B.","quote":"This publication supersedes NIST Special Publication (SP) 800-63B.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"SP 800-63B-4 requires passwords to be salted and hashed with a suitable password hashing scheme.","quote":"Passwords SHALL be salted and hashed using a suitable password hashing scheme.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"SP 800-63B-4 recommends (SHOULD) an approved scheme from SP 800-132 or later NIST guidance.","quote":"An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"SP 800-63B-4 says the cost factor should be as high as practical.","quote":"The chosen cost factor SHOULD be as high as practical without negatively impacting verifier performance.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"SP 800-63B-4 requires a salt of at least 32 bits.","quote":"The salt SHALL be at least 32 bits in length and chosen to minimize salt value collisions among stored hashes (i.e., to prevent multiple subscriber accounts from having the same hashed password).","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"SP 800-63B-4 recommends an extra keyed hashing or encryption step with a secret key known only to the verifier.","quote":"In addition, verifiers SHOULD perform an additional iteration of a keyed hashing or encryption operation using a secret key known only to the verifier.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-3/sp800-63b.html","claim":"The older Revision 3 named PBKDF2 and Balloon as examples and recommended a memory-hard function, language that Revision 4 dropped.","quote":"A memory-hard function SHOULD be used because it increases the cost of an attack.","verified":true,"document":"nist-sp800-63b-3"},{"url":"https://pages.nist.gov/800-63-3/sp800-63b.html","claim":"Revision 3 suggested PBKDF2 iteration counts of typically at least 10,000, far below OWASP's current figure.","quote":"Therefore, the iteration count SHOULD be as large as verification server performance will allow, typically at least 10,000 iterations.","verified":true,"document":"nist-sp800-63b-3"}]
judge unverified quotes
0
judge verdicts
[{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2","E4"]},"nist-current":{"status":"met","support":["E7","E8"]},"nist-revision":{"status":"met","support":["E12"]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E10","E9"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2","E4"]},"nist-current":{"status":"met","support":["E7","E8","E9"]},"nist-revision":{"status":"met","support":["E12"]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E10"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2","E4"]},"nist-current":{"status":"met","support":["E7","E8","E9"]},"nist-revision":{"status":"met","support":["E12"]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E10"]}}]
judge failures
0
grounded weight share
1

Final answer

I've read both documents and am writing up the answer now.

The two sources point the team to the same choice. The current NIST revision requires a salted, hashed password store using a "suitable password hashing scheme" and recommends (SHOULD, not SHALL) a scheme approved by SP 800-132. OWASP says that when FIPS‑140 compliance is required, PBKDF2 should be used. So the team should use PBKDF2‑HMAC‑SHA256 from a FIPS‑140‑validated module, with at least 600,000 iterations, a unique salt for each password, and ideally a secret pepper kept in an HSM. Argon2id is OWASP's general first choice, but OWASP itself sets it aside when FIPS validation is needed.

What OWASP recommends (https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html)

  • Order of preference:

    1. Argon2id, at minimum 19 MiB of memory, 2 iterations and parallelism of 1.
    2. scrypt, if Argon2id isn't available.
    3. bcrypt, for legacy systems only, with a work factor of 10 or more and a 72‑byte input limit.
  • FIPS case: "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256." OWASP gives two reasons: PBKDF2 is permitted by NIST SP 800‑63B‑4, and FIPS‑140‑validated implementations of it exist.
  • Iteration counts: 600,000 for HMAC‑SHA256 and 220,000 for HMAC‑SHA512. HMAC‑SHA1 is "legacy only" and shouldn't be used for new systems.
  • Other advice: use a unique salt per password, and consider a pepper as extra protection. A pepper is a secret key stored separately from the database, such as in a vault or HSM.
  • Minor inconsistencies in the page: the introduction lists bcrypt next to Argon2id and PBKDF2 without qualification, but the summary limits it to legacy systems. It also says "Three hashing algorithms that should be considered" and then describes four.

What NIST SP 800‑63B‑4 requires (https://pages.nist.gov/800-63-4/sp800-63b.html)

  • Status: this is the current revision (the page is dated August 2025) and it "supersedes NIST Special Publication (SP) 800-63B."
  • Mandatory (SHALL):

    • Passwords must be stored in a form that resists offline attacks.
    • They must be "salted and hashed using a suitable password hashing scheme."
    • The salt must be at least 32 bits, and both the salt and the hash must be stored.
  • Recommended (SHOULD):

    • Use an approved scheme from the latest SP 800‑132 or later NIST guidance.
    • Set the cost factor "as high as practical" and raise it over time.
    • Store the scheme and cost factor with each hash so you can migrate later.
    • Apply an extra keyed hashing or encryption step with a secret key. If you do this, the key SHALL be generated by an approved random bit generator and SHALL be stored separately from the hashes. It SHOULD be kept in an HSM or TEE.
  • What it doesn't say: the section I read doesn't name a specific algorithm, and it doesn't mention memory‑hard functions.

How the NIST guidance changed between revisions
The older Revision 3 (https://pages.nist.gov/800-63-3/sp800-63b.html) named PBKDF2 and Balloon as examples. It said "A memory-hard function SHOULD be used" and suggested PBKDF2 iteration counts of "typically at least 10,000 iterations." Revision 4 drops all of that from the corresponding section, so older advice that cites a memory‑hard SHOULD or 10,000 iterations is out of date. OWASP's 600,000‑iteration figure is far stricter than the old NIST number.

How to reconcile them
The sources don't conflict. OWASP's general preference for Argon2id doesn't override NIST, and OWASP itself makes an exception for FIPS. For the federal requirement, the team should:

  1. Use PBKDF2‑HMAC‑SHA256 from a FIPS‑140‑validated cryptographic module.
  2. Set at least 600,000 iterations (OWASP), and keep it "as high as practical" and rising over time (NIST).
  3. Use a unique salt of at least 32 bits for each password, stored with the hash.
  4. Record the algorithm and cost factor with each hash.
  5. Add a secret‑key pepper, generated by an approved random bit generator and kept in an HSM or TEE. Both sources recommend this.

There is one gap. I didn't open SP 800‑132 itself, so the claim that PBKDF2 is the scheme it approves comes from OWASP's statement and the older Revision 3 text, not from SP 800‑132 directly. NIST's "approved scheme" wording is also a SHOULD, so whether non‑approved schemes such as Argon2id are acceptable is a call for your FIPS compliance authority. The NIST text doesn't settle it.

{"evidence": [
{"claim": "OWASP's summary recommends Argon2id first, with minimum parameters.", "quote": "Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP recommends PBKDF2 with HMAC-SHA-256 and at least 600,000 iterations when FIPS-140 compliance is required.", "quote": "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP justifies PBKDF2 for FIPS cases by citing NIST SP 800-63B-4 and FIPS-140-validated implementations.", "quote": "Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP limits bcrypt to legacy systems.", "quote": "The bcrypt password hashing function should only be used for password storage in legacy systems where Argon2 and scrypt are not available.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP suggests a pepper as defense in depth.", "quote": "Consider using a pepper to provide additional defense in depth (though alone, it provides no additional secure characteristics).", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "SP 800-63B-4 supersedes the earlier SP 800-63B.", "quote": "This publication supersedes NIST Special Publication (SP) 800-63B.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "SP 800-63B-4 requires passwords to be salted and hashed with a suitable password hashing scheme.", "quote": "Passwords SHALL be salted and hashed using a suitable password hashing scheme.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "SP 800-63B-4 recommends (SHOULD) an approved scheme from SP 800-132 or later NIST guidance.", "quote": "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "SP 800-63B-4 says the cost factor should be as high as practical.", "quote": "The chosen cost factor SHOULD be as high as practical without negatively impacting verifier performance.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "SP 800-63B-4 requires a salt of at least 32 bits.", "quote": "The salt SHALL be at least 32 bits in length and chosen to minimize salt value collisions among stored hashes (i.e., to prevent multiple subscriber accounts from having the same hashed password).", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "SP 800-63B-4 recommends an extra keyed hashing or encryption step with a secret key known only to the verifier.", "quote": "In addition, verifiers SHOULD perform an additional iteration of a keyed hashing or encryption operation using a secret key known only to the verifier.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "The older Revision 3 named PBKDF2 and Balloon as examples and recommended a memory-hard function, language that Revision 4 dropped.", "quote": "A memory-hard function SHOULD be used because it increases the cost of an attack.", "url": "https://pages.nist.gov/800-63-3/sp800-63b.html"},
{"claim": "Revision 3 suggested PBKDF2 iteration counts of typically at least 10,000, far below OWASP's current figure.", "quote": "Therefore, the iteration count SHOULD be as large as verification server performance will allow, typically at least 10,000 iterations.", "url": "https://pages.nist.gov/800-63-3/sp800-63b.html"}
]}

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.