ScaffoldBench

RunOpus 5.5repeat 2

Single on OWASP vs NIST on password hashing

claude-opus-5-5__single__research-password-hashing__r2

Score

1.000

passed
design ran as designed
Cost
$0.246
+ $0.095 judging
Latency
44.8s
setup 0.0s, grading 6.1s
Turns
9
Tool calls
8
Subagents
0
Tokens in / out
85k / 4.7k
66,372 read from cache

Did the design run?

every check held
  • Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
  • No API failures: heldharness.apiNo model call failed at the API.
  • Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
  • No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
  • Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
  • No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
  • Builtin tools: heldharness.builtin-toolsA check recorded with the run.
  • Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
  • No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
  • Judge answered: heldharness.judgeAt least one grading call returned a usable verdict.
  • Each call was shown the right tools: heldsingle.toolsThe agent was shown exactly the task’s tools, none missing and none extra.

Phases

Main: 0.0s to 44.8s, finished, 8 tool calls, $0.246

Drawn from the phase records: timing and cost per call, no tool calls or messages.

PhaseDurationCostTurnsToolsModel
Main44.8s$0.24698claude-opus-5-5

The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.

Runs you can replay

Record this cell yourself: npm run bench -- run --strategies single --tasks research-password-hashing --models claude-opus-5-5 --repeats 1

Score breakdown

  • passStates that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism.
    1.00weight 0.19

    met, backed by verified evidence E1. “Argon2id, at minimum 19 MiB of memory, 2 iterations and parallelism 1.”

  • passStates OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required.
    1.00weight 0.13

    met, backed by verified evidence E2, E4. “scrypt if Argon2id isn't available.”

  • passStates that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used.
    1.00weight 0.19

    met, backed by verified evidence E7, E8, E9. “use an approved scheme from "the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes"”

  • passStates that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording.
    1.00weight 0.13

    met, backed by verified evidence E14, E15. “Rev. 3 said "A memory-hard function SHOULD be used" and suggested "typically at least 10,000 iterations" for PBKDF2. Rev. 4 drops both”

  • passExplains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply.
    1.00weight 0.25

    met, backed by verified evidence E2, E3. “OWASP and the current NIST SP 800-63B-4 don't really disagree. OWASP prefers Argon2id in general but names PBKDF2 for FIPS cases”

  • passStates at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash.
    1.00weight 0.13

    met, backed by verified evidence E10, E9. “Use a random salt of at least 32 bits for each password.”

Penalties

No penalties were applied.

Scorer metadata

corpus
{"version":2,"snapshot":"2026-10-01","hash":"e12dc326c69a32d91f85f6355dc63e2a7d672b605685a1ffa1582a42bb1c561e","documents":5}
opened documents
["owasp-password-storage","nist-sp800-63b-4","nist-sp800-63b-3"]
opened by role
{"primary":["owasp-password-storage","nist-sp800-63b-4"],"superseded":["nist-sp800-63b-3"]}
displayed share
{"owasp-password-storage":0.8205,"nist-sp800-63b-4":0.0607,"nist-sp800-63b-3":0.0456,"rfc9106":0.009,"owasp-authentication":0.007}
cited count
3
verified count
3
hallucinated count
0
citation integrity
1
hallucinated urls
[]
evidence block
true
evidence block repaired
false
evidence count
15
verified quote count
15
verified quote rate
1
evidence
[{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP's general recommendation is Argon2id with minimum parameters of 19 MiB memory, 2 iterations, parallelism 1.","quote":"Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP recommends PBKDF2-HMAC-SHA-256 with at least 600,000 iterations when FIPS-140 compliance is required.","quote":"If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP says PBKDF2 should be preferred where NIST 800-63B-4 / FIPS-140 validation is required.","quote":"Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP treats bcrypt as legacy-only.","quote":"The bcrypt password hashing function should only be used for password storage in legacy systems where Argon2 and scrypt are not available.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP presents a pepper as an optional consideration.","quote":"Consider using a pepper to provide additional defense in depth (though alone, it provides no additional secure characteristics).","verified":true,"document":"owasp-password-storage"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"SP 800-63B-4 supersedes the earlier SP 800-63B.","quote":"This publication supersedes NIST Special Publication (SP) 800-63B.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"SP 800-63B-4 mandates salting and hashing with a suitable password hashing scheme.","quote":"Passwords SHALL be salted and hashed using a suitable password hashing scheme.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"SP 800-63B-4 recommends an approved scheme from SP 800-132 or updated NIST guidance.","quote":"An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"SP 800-63B-4 asks for a cost factor as high as practical, raised over time.","quote":"The chosen cost factor SHOULD be as high as practical without negatively impacting verifier performance.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"SP 800-63B-4 requires a salt of at least 32 bits.","quote":"The salt SHALL be at least 32 bits in length and chosen to minimize salt value collisions among stored hashes (i.e., to prevent multiple subscriber accounts from having the same hashed password).","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"SP 800-63B-4 recommends storing the scheme and cost factor to allow migration.","quote":"A reference to the password hashing scheme used, including the cost factor, SHOULD be stored for each password to allow migration to new algorithms and work factors.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"SP 800-63B-4 recommends an additional keyed hashing step with a secret key (a pepper).","quote":"In addition, verifiers SHOULD perform an additional iteration of a keyed hashing or encryption operation using a secret key known only to the verifier.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"Under SP 800-63B-4, the pepper key must be stored separately from the hashes.","quote":"The secret key value SHALL be stored separately from the hashed passwords.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-3/sp800-63b.html","claim":"The superseded Rev. 3 recommended a memory-hard function, wording not found in Rev. 4's password hashing section.","quote":"A memory-hard function SHOULD be used because it increases the cost of an attack.","verified":true,"document":"nist-sp800-63b-3"},{"url":"https://pages.nist.gov/800-63-3/sp800-63b.html","claim":"The superseded Rev. 3 suggested typically at least 10,000 PBKDF2 iterations.","quote":"Therefore, the iteration count SHOULD be as large as verification server performance will allow, typically at least 10,000 iterations.","verified":true,"document":"nist-sp800-63b-3"}]
judge unverified quotes
0
judge verdicts
[{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2","E4"]},"nist-current":{"status":"met","support":["E7","E8","E9"]},"nist-revision":{"status":"met","support":["E14","E15"]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E10","E9"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2","E4"]},"nist-current":{"status":"met","support":["E7","E8","E9"]},"nist-revision":{"status":"met","support":["E14","E15"]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E10","E9"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2","E4"]},"nist-current":{"status":"met","support":["E7","E8","E9"]},"nist-revision":{"status":"met","support":["E14","E15"]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E10","E9"]}}]
judge failures
0
grounded weight share
1

Final answer

Recommendation: use PBKDF2 with HMAC-SHA-256 and at least 600,000 iterations, through a FIPS 140-validated crypto module. Give each password a unique random salt of at least 32 bits, and add a secret-keyed HMAC "pepper" kept in an HSM or similar. OWASP and the current NIST SP 800-63B-4 don't really disagree. OWASP prefers Argon2id in general but names PBKDF2 for FIPS cases, and that is also the scheme NIST's text points to.

What OWASP says (https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html)

  • Order of preference:

    1. Argon2id, at minimum 19 MiB of memory, 2 iterations and parallelism 1.
    2. scrypt if Argon2id isn't available.
    3. bcrypt only for legacy systems (work factor 10 or more, 72-byte input limit).
  • FIPS case: "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256." OWASP explains that PBKDF2 "is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations," so it "should be the preferred algorithm when these are required."
  • PBKDF2 iteration counts: 600,000 for HMAC-SHA256 and 220,000 for HMAC-SHA512. HMAC-SHA1 is "legacy only."
  • Salt and pepper: a unique salt is required. A pepper is something to "consider" as extra defence.

What NIST SP 800-63B-4 says (https://pages.nist.gov/800-63-4/sp800-63b.html, dated 26 Aug 2025; it says it supersedes the earlier SP 800-63B)

  • Mandatory (SHALL):

    • passwords are stored in a form resistant to offline attacks, salted and hashed with "a suitable password hashing scheme";
    • the salt is at least 32 bits;
    • both the salt and the hash are stored.
  • Recommended (SHOULD):

    • use an approved scheme from "the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes";
    • set the cost factor as high as practical and raise it over time;
    • record the scheme and cost factor with each hash;
    • do one more keyed hashing or encryption step with a secret key, which is a pepper. If used, that key SHALL come from an approved random bit generator and SHALL be stored separately from the hashes, ideally in an HSM or TEE.
  • What it doesn't say: it gives no iteration count and doesn't name Argon2, scrypt or bcrypt.
  • Change from Rev. 3 (https://pages.nist.gov/800-63-3/sp800-63b.html): Rev. 3 said "A memory-hard function SHOULD be used" and suggested "typically at least 10,000 iterations" for PBKDF2. Rev. 4 drops both, so the 10,000 figure is out of date.

How to reconcile them

  1. Algorithm. PBKDF2 meets OWASP's FIPS recommendation and the scheme NIST points to through SP 800-132. Argon2id is OWASP's general first choice, but it isn't what NIST's text points to. I didn't open SP 800-132 itself or any FIPS 140 validation listing, so what PBKDF2's approval and validated implementations rest on is OWASP's statement plus NIST's reference.
  2. Cost. NIST gives no number, so OWASP's 600,000 iterations (HMAC-SHA-256) is a reasonable concrete starting point. Then follow NIST: as high as performance allows, raised over time.
  3. Pepper. NIST is stronger here: a SHOULD, versus OWASP's "consider." For a federal context, treat the keyed step as expected and follow NIST's rules on generating and storing the key.
  4. Storage and migration. Use a random salt of at least 32 bits for each password. Store the algorithm and parameters with every hash, so you can raise the work factor or move to a newer NIST-approved scheme later. OWASP describes doing this by re-hashing when the user next logs in.
  5. Without the FIPS constraint, OWASP would recommend Argon2id instead.
{"evidence": [
{"claim": "OWASP's general recommendation is Argon2id with minimum parameters of 19 MiB memory, 2 iterations, parallelism 1.", "quote": "Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP recommends PBKDF2-HMAC-SHA-256 with at least 600,000 iterations when FIPS-140 compliance is required.", "quote": "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP says PBKDF2 should be preferred where NIST 800-63B-4 / FIPS-140 validation is required.", "quote": "Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP treats bcrypt as legacy-only.", "quote": "The bcrypt password hashing function should only be used for password storage in legacy systems where Argon2 and scrypt are not available.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP presents a pepper as an optional consideration.", "quote": "Consider using a pepper to provide additional defense in depth (though alone, it provides no additional secure characteristics).", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "SP 800-63B-4 supersedes the earlier SP 800-63B.", "quote": "This publication supersedes NIST Special Publication (SP) 800-63B.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "SP 800-63B-4 mandates salting and hashing with a suitable password hashing scheme.", "quote": "Passwords SHALL be salted and hashed using a suitable password hashing scheme.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "SP 800-63B-4 recommends an approved scheme from SP 800-132 or updated NIST guidance.", "quote": "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "SP 800-63B-4 asks for a cost factor as high as practical, raised over time.", "quote": "The chosen cost factor SHOULD be as high as practical without negatively impacting verifier performance.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "SP 800-63B-4 requires a salt of at least 32 bits.", "quote": "The salt SHALL be at least 32 bits in length and chosen to minimize salt value collisions among stored hashes (i.e., to prevent multiple subscriber accounts from having the same hashed password).", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "SP 800-63B-4 recommends storing the scheme and cost factor to allow migration.", "quote": "A reference to the password hashing scheme used, including the cost factor, SHOULD be stored for each password to allow migration to new algorithms and work factors.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "SP 800-63B-4 recommends an additional keyed hashing step with a secret key (a pepper).", "quote": "In addition, verifiers SHOULD perform an additional iteration of a keyed hashing or encryption operation using a secret key known only to the verifier.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "Under SP 800-63B-4, the pepper key must be stored separately from the hashes.", "quote": "The secret key value SHALL be stored separately from the hashed passwords.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "The superseded Rev. 3 recommended a memory-hard function, wording not found in Rev. 4's password hashing section.", "quote": "A memory-hard function SHOULD be used because it increases the cost of an attack.", "url": "https://pages.nist.gov/800-63-3/sp800-63b.html"},
{"claim": "The superseded Rev. 3 suggested typically at least 10,000 PBKDF2 iterations.", "quote": "Therefore, the iteration count SHOULD be as large as verification server performance will allow, typically at least 10,000 iterations.", "url": "https://pages.nist.gov/800-63-3/sp800-63b.html"}
]}

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.