ScaffoldBench

RunSonnet 5.5repeat 2

Plan-execute-verify on OWASP vs NIST on password hashing

claude-sonnet-5-5__plan-execute-verify__research-password-hashing__r2

Score

0.875

wrong answer
design ran as designed
Cost
$0.249
+ $0.093 judging
Latency
54.5s
setup 0.0s, grading 5.5s
Turns
20
Tool calls
15
Subagents
0
Tokens in / out
147k / 7.1k
101,355 read from cache

Did the design run?

every check held
  • Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
  • No API failures: heldharness.apiNo model call failed at the API.
  • Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
  • No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
  • Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
  • No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
  • Builtin tools: heldharness.builtin-toolsA check recorded with the run.
  • Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
  • No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
  • Judge answered: heldharness.judgeAt least one grading call returned a usable verdict.
  • The plan was a plan: heldplan-execute-verify.planThe planner’s reply was read through its schema and held at least one step.
  • Every verdict was read: heldplan-execute-verify.verdictsEvery review round ended with a verdict read from the schema reply, not a default.
  • The loop did what its verdicts said: heldplan-execute-verify.loopEvery verdict that called for a revision got one, and nothing was revised without such a verdict.
  • Each call was shown the right tools: heldplan-execute-verify.toolsThe planner was shown no tools, the verifier the read-only ones, and the executor and reviser the full toolset.

Phases

Plan: 0.0s to 15.4s, finished, 0 tool calls, $0.020
Execute: 15.4s to 44.8s, finished, 9 tool calls, $0.150
Verify: 44.8s to 54.5s, finished, 6 tool calls, $0.080

Drawn from the phase records: timing and cost per call, no tool calls or messages.

PhaseDurationCostTurnsToolsModel
Plan15.4s$0.02020claude-sonnet-5-5
Execute29.4s$0.150109claude-sonnet-5-5
Verify9.8s$0.08086claude-sonnet-5-5

The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.

Runs you can replay

Record this cell yourself: npm run bench -- run --strategies plan-execute-verify --tasks research-password-hashing --models claude-sonnet-5-5 --repeats 1

Score breakdown

  • passStates that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism.
    1.00weight 0.19

    met, backed by verified evidence E1. “Argon2id is first: "Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism."”

  • passStates OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required.
    1.00weight 0.13

    met, backed by verified evidence E2. “scrypt is the fallback if Argon2id is unavailable.”

  • passStates that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used.
    1.00weight 0.19

    met, backed by verified evidence E7, E8. “"An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used." This is a SHOULD, not a SHALL.”

  • failStates that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording.
    0.00weight 0.13

    Not stated.

  • passExplains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply.
    1.00weight 0.25

    met, backed by verified evidence E2, E3. “Use PBKDF2-HMAC-SHA-256 with at least 600,000 iterations, as OWASP recommends.”

  • passStates at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash.
    1.00weight 0.13

    met, backed by verified evidence E7, E9, E10, E12. “The sheet recommends a unique random salt per password”

Penalties

No penalties were applied.

Scorer metadata

corpus
{"version":2,"snapshot":"2026-10-01","hash":"e12dc326c69a32d91f85f6355dc63e2a7d672b605685a1ffa1582a42bb1c561e","documents":5}
opened documents
["owasp-password-storage","nist-sp800-63b-4"]
opened by role
{"primary":["owasp-password-storage","nist-sp800-63b-4"]}
displayed share
{"owasp-password-storage":1,"nist-sp800-63b-4":0.0724,"rfc9106":0.0076,"nist-sp800-63b-3":0.0014,"owasp-authentication":0.0068}
cited count
2
verified count
2
hallucinated count
0
citation integrity
1
hallucinated urls
[]
evidence block
true
evidence block repaired
false
evidence count
12
verified quote count
12
verified quote rate
1
evidence
[{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP's primary recommendation is Argon2id with a minimum configuration.","quote":"Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP recommends PBKDF2 with HMAC-SHA-256 and at least 600,000 iterations when FIPS-140 compliance is required.","quote":"If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP states PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, so it is preferred when FIPS is required.","quote":"Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP gives a lower iteration count for PBKDF2-HMAC-SHA512.","quote":"PBKDF2-HMAC-SHA512: 220,000 iterations","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP advises an optional pepper stored separately from the database.","quote":"Unlike a password salt, the pepper should not be public and should not be stored along with the generated hash.","verified":true,"document":"owasp-password-storage"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST SP 800-63B-4 supersedes the earlier SP 800-63B and is dated 26 Aug 2025.","quote":"This publication supersedes NIST Special Publication (SP) 800-63B.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST requires passwords to be salted and hashed with a suitable password hashing scheme.","quote":"Passwords SHALL be salted and hashed using a suitable password hashing scheme.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST defers to an approved scheme in SP 800-132 or updated NIST guidance, as a SHOULD, without naming an algorithm in this passage.","quote":"An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST requires a salt of at least 32 bits and storage of both salt and hash.","quote":"The salt SHALL be at least 32 bits in length and chosen to minimize salt value collisions among stored hashes","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST recommends storing the scheme and cost factor to allow migration.","quote":"A reference to the password hashing scheme used, including the cost factor, SHOULD be stored for each password to allow migration to new algorithms and work factors.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST recommends an additional keyed hash or encryption step with a secret key stored separately, ideally in an HSM or TEE.","quote":"In addition, verifiers SHOULD perform an additional iteration of a keyed hashing or encryption operation using a secret key known only to the verifier.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST says the cost factor should be as high as practical and increased over time.","quote":"The chosen cost factor SHOULD be as high as practical without negatively impacting verifier performance.","verified":true,"document":"nist-sp800-63b-4"}]
judge unverified quotes
0
judge verdicts
[{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2"]},"nist-current":{"status":"met","support":["E7","E8"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E7","E9","E10","E12"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2"]},"nist-current":{"status":"met","support":["E6","E7","E8","E10","E12"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E7","E9","E12"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2"]},"nist-current":{"status":"met","support":["E6","E7","E8","E10","E12"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E7","E9","E12"]}}]
judge failures
0
grounded weight share
0.875 (88%)

Final answer

Short answer: Use PBKDF2-HMAC-SHA-256 with a FIPS 140-validated module. OWASP recommends Argon2id in general but points to PBKDF2 when FIPS compliance is required. The current NIST SP 800-63B (Revision 4) sets requirements on salting, cost factor and storage, and does not name a required algorithm in the passage I read.

What OWASP recommends

The OWASP Password Storage Cheat Sheet is at https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html. It gives an ordered list:

  • Argon2id is first: "Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism."
  • scrypt is the fallback if Argon2id is unavailable.
  • bcrypt is for legacy systems only, with a work factor of 10 or more and a 72-byte password limit.
  • For FIPS, it says: "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256."

In its PBKDF2 section, OWASP gives these iteration counts:

  • PBKDF2-HMAC-SHA256: 600,000 iterations (recommended).
  • PBKDF2-HMAC-SHA512: 220,000 iterations.
  • PBKDF2-HMAC-SHA1: 1,400,000 iterations, legacy only.

OWASP also states that "Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required."

The sheet recommends a unique random salt per password, and says libraries usually generate it for you. It also suggests an optional secret pepper, stored separately from the database (for example in a secrets vault or HSM). It recommends storing the algorithm and work factor with each hash so you can migrate later.

What NIST SP 800-63B requires

The page at https://pages.nist.gov/800-63-4/sp800-63b.html is titled "NIST SP 800-63B-4". It is dated 26 Aug 2025, and its abstract says "This publication supersedes NIST Special Publication (SP) 800-63B." So Revision 4 is the current version, and it replaces Revision 3. I did not check any NIST status page beyond this one.

The password-storage requirements in section 3.1.1.2 are:

  • "Passwords SHALL be salted and hashed using a suitable password hashing scheme."
  • The cost factor SHOULD be as high as practical without hurting verifier performance, and SHOULD be increased over time.
  • "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used." This is a SHOULD, not a SHALL.
  • "The salt SHALL be at least 32 bits in length", chosen to minimize collisions among stored hashes.
  • The salt and hash SHALL both be stored. A reference to the scheme and cost factor SHOULD be stored too, so you can migrate to new algorithms and work factors.
  • Verifiers SHOULD add a keyed hash or encryption step using a secret key. That key SHALL come from an approved random bit generator and SHALL be stored separately from the hashes. NIST says it SHOULD be kept in a hardware-protected area such as an HSM or TEE. This is NIST's equivalent of a pepper.

In the passage I read, 800-63B-4 does not name PBKDF2 or Argon2id. It defers to SP 800-132 or future NIST guidance. I did not open SP 800-132, so I cannot confirm from a source I read that PBKDF2 is the scheme it approves. OWASP asserts this ("permitted by NIST SP 800-63B-4"), but it is OWASP's statement, not NIST's. I also did not open any source on whether Argon2id is FIPS-approved, so I make no claim either way.

How to reconcile them (these steps are my inference, not stated by either source)

  1. Do not choose Argon2id, scrypt or bcrypt as the primary scheme if the system must use FIPS-approved algorithms. OWASP itself points to PBKDF2 for that case, and NIST's "approved scheme" language points toward SP 800-132.
  2. Use PBKDF2-HMAC-SHA-256 with at least 600,000 iterations, as OWASP recommends. Use a unique random salt per password. 16 bytes is a common choice and comfortably exceeds NIST's 32-bit minimum. Run it in a FIPS 140-validated cryptographic module.
  3. Add NIST's keyed step: an HMAC with a secret key from an approved random bit generator, stored in an HSM or secrets vault. This also matches OWASP's pepper advice.
  4. Store the scheme and cost factor with each hash. Re-tune the iteration count as hardware improves, and plan to move to Argon2id if your FIPS scope later allows it. Both sources support this kind of migration.
  5. Confirm the details with your compliance or authorizing officer. FIPS applicability depends on the system's authorization boundary, and I did not verify that against any source.
{"evidence": [
{"claim": "OWASP's primary recommendation is Argon2id with a minimum configuration.", "quote": "Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP recommends PBKDF2 with HMAC-SHA-256 and at least 600,000 iterations when FIPS-140 compliance is required.", "quote": "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP states PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, so it is preferred when FIPS is required.", "quote": "Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP gives a lower iteration count for PBKDF2-HMAC-SHA512.", "quote": "PBKDF2-HMAC-SHA512: 220,000 iterations", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP advises an optional pepper stored separately from the database.", "quote": "Unlike a password salt, the pepper should not be public and should not be stored along with the generated hash.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "NIST SP 800-63B-4 supersedes the earlier SP 800-63B and is dated 26 Aug 2025.", "quote": "This publication supersedes NIST Special Publication (SP) 800-63B.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST requires passwords to be salted and hashed with a suitable password hashing scheme.", "quote": "Passwords SHALL be salted and hashed using a suitable password hashing scheme.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST defers to an approved scheme in SP 800-132 or updated NIST guidance, as a SHOULD, without naming an algorithm in this passage.", "quote": "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST requires a salt of at least 32 bits and storage of both salt and hash.", "quote": "The salt SHALL be at least 32 bits in length and chosen to minimize salt value collisions among stored hashes", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST recommends storing the scheme and cost factor to allow migration.", "quote": "A reference to the password hashing scheme used, including the cost factor, SHOULD be stored for each password to allow migration to new algorithms and work factors.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST recommends an additional keyed hash or encryption step with a secret key stored separately, ideally in an HSM or TEE.", "quote": "In addition, verifiers SHOULD perform an additional iteration of a keyed hashing or encryption operation using a secret key known only to the verifier.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST says the cost factor should be as high as practical and increased over time.", "quote": "The chosen cost factor SHOULD be as high as practical without negatively impacting verifier performance.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"}
]}

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.