ScaffoldBench

RunSonnet 5.5repeat 2

Single on OWASP vs NIST on password hashing

claude-sonnet-5-5__single__research-password-hashing__r2

Score

0.875

wrong answer
design ran as designed
Cost
$0.110
+ $0.089 judging
Latency
29.4s
setup 0.0s, grading 4.7s
Turns
9
Tool calls
8
Subagents
0
Tokens in / out
51k / 4.2k
34,945 read from cache

Did the design run?

every check held
  • Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
  • No API failures: heldharness.apiNo model call failed at the API.
  • Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
  • No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
  • Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
  • No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
  • Builtin tools: heldharness.builtin-toolsA check recorded with the run.
  • Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
  • No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
  • Judge answered: heldharness.judgeAt least one grading call returned a usable verdict.
  • Each call was shown the right tools: heldsingle.toolsThe agent was shown exactly the task’s tools, none missing and none extra.

Phases

Main: 0.0s to 29.4s, finished, 8 tool calls, $0.110

Drawn from the phase records: timing and cost per call, no tool calls or messages.

PhaseDurationCostTurnsToolsModel
Main29.4s$0.11098claude-sonnet-5-5

The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.

Runs you can replay

Record this cell yourself: npm run bench -- run --strategies single --tasks research-password-hashing --models claude-sonnet-5-5 --repeats 1

Score breakdown

  • passStates that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism.
    1.00weight 0.19

    met, backed by verified evidence E2. “Its default advice is Argon2id with at least 19 MiB of memory, 2 iterations and 1 degree of parallelism.”

  • passStates OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required.
    1.00weight 0.13

    met, backed by verified evidence E1, E4. “If Argon2id isn't available, it says to use scrypt.”

  • passStates that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used.
    1.00weight 0.19

    met, backed by verified evidence E6, E7, E8. “An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.”

  • failStates that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording.
    0.00weight 0.13

    Not stated.

  • passExplains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply.
    1.00weight 0.25

    met, backed by verified evidence E1, E3. “If the service really must meet FIPS or federal requirements, use OWASP's FIPS branch: PBKDF2 with HMAC-SHA-256 and at least 600,000 iterations.”

  • passStates at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash.
    1.00weight 0.13

    met, backed by verified evidence E6, E8, E9. “Use a unique random salt of at least 32 bits”

Penalties

No penalties were applied.

Scorer metadata

corpus
{"version":2,"snapshot":"2026-10-01","hash":"e12dc326c69a32d91f85f6355dc63e2a7d672b605685a1ffa1582a42bb1c561e","documents":5}
opened documents
["owasp-password-storage","nist-sp800-63b-4"]
opened by role
{"primary":["owasp-password-storage","nist-sp800-63b-4"]}
displayed share
{"owasp-password-storage":1,"nist-sp800-63b-4":0.0602}
cited count
2
verified count
2
hallucinated count
0
citation integrity
1
hallucinated urls
[]
evidence block
true
evidence block repaired
false
evidence count
11
verified quote count
11
verified quote rate
1
evidence
[{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP recommends PBKDF2 with 600,000+ iterations and HMAC-SHA-256 when FIPS-140 compliance is required","quote":"If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP's default recommendation is Argon2id with minimum parameters","quote":"Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP justifies PBKDF2 for FIPS by asserting NIST permits it and FIPS-validated implementations exist","quote":"Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP limits bcrypt to legacy systems","quote":"The bcrypt password hashing function should only be used for password storage in legacy systems where Argon2 and scrypt are not available.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP lists PBKDF2 iteration counts per inner hash and marks SHA-1 as legacy only","quote":"PBKDF2-HMAC-SHA256: 600,000 iterations (recommended)","verified":true,"document":"owasp-password-storage"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST 800-63B-4 requires salted hashing with a suitable password hashing scheme","quote":"Passwords SHALL be salted and hashed using a suitable password hashing scheme.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST defers the choice of approved scheme to SP 800-132 or updated NIST guidance, without naming an algorithm","quote":"An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST says the cost factor should be as high as practical and increased over time","quote":"The chosen cost factor SHOULD be as high as practical without negatively impacting verifier performance.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST sets salt length and storage requirements","quote":"The salt SHALL be at least 32 bits in length and chosen to minimize salt value collisions among stored hashes","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST says to store the scheme and cost factor with each password to allow migration","quote":"A reference to the password hashing scheme used, including the cost factor, SHOULD be stored for each password to allow migration to new algorithms and work factors.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST recommends an additional keyed hash/encryption step with a secret key stored separately, ideally in an HSM or TEE","quote":"In addition, verifiers SHOULD perform an additional iteration of a keyed hashing or encryption operation using a secret key known only to the verifier.","verified":true,"document":"nist-sp800-63b-4"}]
judge unverified quotes
0
judge verdicts
[{"owasp-argon2id":{"status":"met","support":["E2"]},"owasp-conditions":{"status":"met","support":["E1","E4"]},"nist-current":{"status":"met","support":["E6","E7","E8"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E1","E3"]},"common-ground":{"status":"met","support":["E6","E8","E9"]}},{"owasp-argon2id":{"status":"met","support":["E2"]},"owasp-conditions":{"status":"met","support":["E1","E4"]},"nist-current":{"status":"met","support":["E6","E7","E8"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E1","E3"]},"common-ground":{"status":"met","support":["E6","E8","E9"]}},{"owasp-argon2id":{"status":"met","support":["E2"]},"owasp-conditions":{"status":"met","support":["E1","E4"]},"nist-current":{"status":"met","support":["E6","E7"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E1","E3"]},"common-ground":{"status":"met","support":["E6","E8"]}}]
judge failures
0
grounded weight share
0.875 (88%)

Final answer

OWASP recommends Argon2id in general and PBKDF2 where FIPS-140 is required. NIST SP 800-63B-4 doesn't name an algorithm. It requires salted hashing with an approved scheme and defers the choice to SP 800-132. So a FIPS-bound team should use PBKDF2-HMAC-SHA-256 with a high iteration count. I read the NIST text on the 26 Aug 2025 page of the final SP 800-63B-4, which supersedes the earlier 800-63B.

OWASP Password Storage Cheat Sheet

  • Its default advice is Argon2id with at least 19 MiB of memory, 2 iterations and 1 degree of parallelism. Its detailed list also gives equivalent trade-offs, such as 46 MiB with 1 iteration.
  • If Argon2id isn't available, it says to use scrypt. The minimum is a CPU/memory cost of 2^17, a block size of 8 and a parallelization of 1.
  • bcrypt is for legacy systems only, with a work factor of 10 or more and a 72-byte password limit.
  • For FIPS, it says: "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256."
  • It gives other PBKDF2 iteration counts: 220,000 for HMAC-SHA-512, and 1,400,000 for HMAC-SHA-1. It marks SHA-1 as legacy only, not for new systems.
  • It recommends a unique salt per password. It also suggests an optional pepper stored separately, for example in a vault or HSM.
  • Its stated rationale for PBKDF2 is: "Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required."

NIST SP 800-63B-4 (current revision)

  • Verifiers must store passwords so they resist offline attacks: "Passwords SHALL be salted and hashed using a suitable password hashing scheme."
  • A scheme takes a password, a salt and a cost factor. The cost factor "SHOULD be as high as practical without negatively impacting verifier performance" and should be increased over time.
  • It says: "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used."
  • The salt must be at least 32 bits and chosen to minimize collisions between stored hashes. The salt and the hash must both be stored.
  • A reference to the scheme and its cost factor should be stored with each password, so you can migrate later.
  • Verifiers "SHOULD perform an additional iteration of a keyed hashing or encryption operation using a secret key known only to the verifier". The key must come from an approved random bit generator. It must be stored separately from the hashes, ideally in an HSM or TEE. This is NIST's version of a pepper.
  • A search of the page found no occurrence of "PBKDF" or "key derivation function". SP 800-63B-4 therefore doesn't name PBKDF2 or Argon2id, and it doesn't set an iteration count.

Where the sources differ

  • OWASP says NIST SP 800-63B-4 "permits" PBKDF2. The 800-63B-4 text I read doesn't name it. It only points to the approved schemes in SP 800-132, so OWASP's statement is an inference. I did not open SP 800-132, so I haven't confirmed that it lists PBKDF2.
  • OWASP's general first choice, Argon2id, is not something 800-63B-4 prohibits or approves by name. NIST approval comes only through SP 800-132 or updated NIST guidance.

How to reconcile them

  1. If the service really must meet FIPS or federal requirements, use OWASP's FIPS branch: PBKDF2 with HMAC-SHA-256 and at least 600,000 iterations. Use a FIPS-140-validated cryptographic module. Argon2id is OWASP's general default, but it doesn't fit the FIPS case. It is not named in 800-63B-4, and I haven't confirmed it is approved under SP 800-132.
  2. Meet the NIST "SHALL" requirements. Use a unique random salt of at least 32 bits (use 16 bytes or more in practice). Store the salt, the hash, and the scheme and cost factor for each password.
  3. Add the keyed hash or pepper. NIST says SHOULD, and OWASP recommends it as defense in depth. Keep the key in an HSM or TEE, separate from the database.
  4. Tune the iteration count to your server. NIST says as high as practical, and OWASP gives a floor of 600,000 for SHA-256. OWASP also warns that very high work factors open a denial-of-service risk, and suggests hashing take under a second. Raise the count over time, and re-hash at the next login.
  5. If FIPS isn't actually mandatory for this service, OWASP's preferred Argon2id is the stronger choice. Check with your compliance owner which regime applies. The stored scheme metadata lets you migrate later.
{"evidence": [
{"claim": "OWASP recommends PBKDF2 with 600,000+ iterations and HMAC-SHA-256 when FIPS-140 compliance is required", "quote": "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP's default recommendation is Argon2id with minimum parameters", "quote": "Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP justifies PBKDF2 for FIPS by asserting NIST permits it and FIPS-validated implementations exist", "quote": "Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP limits bcrypt to legacy systems", "quote": "The bcrypt password hashing function should only be used for password storage in legacy systems where Argon2 and scrypt are not available.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP lists PBKDF2 iteration counts per inner hash and marks SHA-1 as legacy only", "quote": "PBKDF2-HMAC-SHA256: 600,000 iterations (recommended)", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "NIST 800-63B-4 requires salted hashing with a suitable password hashing scheme", "quote": "Passwords SHALL be salted and hashed using a suitable password hashing scheme.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST defers the choice of approved scheme to SP 800-132 or updated NIST guidance, without naming an algorithm", "quote": "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST says the cost factor should be as high as practical and increased over time", "quote": "The chosen cost factor SHOULD be as high as practical without negatively impacting verifier performance.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST sets salt length and storage requirements", "quote": "The salt SHALL be at least 32 bits in length and chosen to minimize salt value collisions among stored hashes", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST says to store the scheme and cost factor with each password to allow migration", "quote": "A reference to the password hashing scheme used, including the cost factor, SHOULD be stored for each password to allow migration to new algorithms and work factors.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST recommends an additional keyed hash/encryption step with a secret key stored separately, ideally in an HSM or TEE", "quote": "In addition, verifiers SHOULD perform an additional iteration of a keyed hashing or encryption operation using a secret key known only to the verifier.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"}
]}

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.