ScaffoldBench

RunSonnet 5.5repeat 2

Propose-select on OWASP vs NIST on password hashing

claude-sonnet-5-5__propose-select__research-password-hashing__r2

Score

0.875

wrong answer
design ran as designed
Cost
$0.424
+ $0.085 judging
Latency
54.3s
setup 0.0s, grading 4.6s
Turns
25
Tool calls
20
Subagents
3
Tokens in / out
170k / 18k
103,034 read from cache

Did the design run?

every check held
  • Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
  • No API failures: heldharness.apiNo model call failed at the API.
  • Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
  • No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
  • Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
  • No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
  • Builtin tools: heldharness.builtin-toolsA check recorded with the run.
  • Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
  • No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
  • Judge answered: heldharness.judgeAt least one grading call returned a usable verdict.
  • All three proposers ran: heldpropose-select.proposalsThree proposers ran in parallel, each from its own angle, and none was starved while another ran.
  • The selector chose: heldpropose-select.selectionThe selector picked a proposal by a valid number, read from its schema reply and not a default.
  • Each call was shown the right tools: heldpropose-select.toolsEach proposer was shown the read-only tools, the selector none, and the executor the full toolset.

Phases

Proposal 1 (policy-first): 0.0s to 31.0s, finished, 8 tool calls, $0.116
Proposal 2 (state-first): 0.0s to 28.1s, finished, 5 tool calls, $0.101
Proposal 3 (outcome-first): 0.0s to 30.4s, finished, 7 tool calls, $0.110
Select: 31.0s to 54.3s, finished, 0 tool calls, $0.098

Drawn from the phase records: timing and cost per call, no tool calls or messages.

PhaseDurationCostTurnsToolsModel
Proposal 1 (policy-first)31.0s$0.11698claude-sonnet-5-5
Proposal 2 (state-first)28.1s$0.10165claude-sonnet-5-5
Proposal 3 (outcome-first)30.4s$0.11087claude-sonnet-5-5
Select23.2s$0.09820claude-sonnet-5-5

The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.

Runs you can replay

Record this cell yourself: npm run bench -- run --strategies propose-select --tasks research-password-hashing --models claude-sonnet-5-5 --repeats 1

Score breakdown

  • passStates that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism.
    1.00weight 0.19

    met, backed by verified evidence E2. “Argon2id first, with at least 19 MiB of memory, 2 iterations and parallelism 1.”

  • passStates OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required.
    1.00weight 0.13

    met, backed by verified evidence E3. “scrypt if Argon2id is unavailable.”

  • passStates that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used.
    1.00weight 0.19

    met, backed by verified evidence E6, E7, E9, E12. “An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.”

  • failStates that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording.
    0.00weight 0.13

    Not stated.

  • passExplains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply.
    1.00weight 0.25

    met, backed by verified evidence E3, E4. “Use PBKDF2-HMAC-SHA-256 with at least 600,000 iterations for the FIPS-scoped service. This follows OWASP's FIPS exception and fits NIST's "approved scheme" language.”

  • passStates at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash.
    1.00weight 0.13

    met, backed by verified evidence E1, E6, E8, E9. “The cost factor SHOULD be as high as practical without hurting verifier performance, and SHOULD be increased over time.”

Penalties

No penalties were applied.

Scorer metadata

corpus
{"version":2,"snapshot":"2026-10-01","hash":"e12dc326c69a32d91f85f6355dc63e2a7d672b605685a1ffa1582a42bb1c561e","documents":5}
opened documents
["owasp-password-storage","nist-sp800-63b-4"]
opened by role
{"primary":["owasp-password-storage","nist-sp800-63b-4"]}
displayed share
{"owasp-password-storage":1,"nist-sp800-63b-4":0.0624}
cited count
2
verified count
2
hallucinated count
0
citation integrity
1
hallucinated urls
[]
evidence block
true
evidence block repaired
false
evidence count
12
verified quote count
12
verified quote rate
1
evidence
[{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP's general rule: passwords must be protected with strong, slow hashing algorithms (Argon2id, bcrypt, PBKDF2).","quote":"Instead, they must be protected using strong, slow hashing algorithms such as Argon2id, bcrypt, or PBKDF2.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP's primary recommendation is Argon2id with minimum parameters of 19 MiB, 2 iterations and parallelism 1.","quote":"Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"For FIPS-140 compliance OWASP recommends PBKDF2 with HMAC-SHA-256 and at least 600,000 iterations.","quote":"If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP says PBKDF2 should be preferred when FIPS-validated implementations are needed, and states that NIST SP 800-63B-4 permits it.","quote":"Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP gives PBKDF2-HMAC-SHA256 at 600,000 iterations as the recommended setting.","quote":"PBKDF2-HMAC-SHA256: 600,000 iterations (recommended)","verified":true,"document":"owasp-password-storage"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST SP 800-63B-4 requires passwords to be salted and hashed with a suitable password hashing scheme.","quote":"Passwords SHALL be salted and hashed using a suitable password hashing scheme.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST defers to approved schemes in SP 800-132 or updated NIST guidance, as a SHOULD, and does not name an algorithm in this text.","quote":"An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST requires a salt of at least 32 bits, with both salt and hash stored.","quote":"The salt SHALL be at least 32 bits in length and chosen to minimize salt value collisions among stored hashes","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST says the cost factor should be as high as practical and increased over time.","quote":"The chosen cost factor SHOULD be as high as practical without negatively impacting verifier performance. It SHOULD be increased over time to account for increases in computing performance.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST recommends storing a reference to the scheme and cost factor to allow migration.","quote":"A reference to the password hashing scheme used, including the cost factor, SHOULD be stored for each password to allow migration to new algorithms and work factors.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST recommends an additional keyed hash or encryption step using a secret key known only to the verifier.","quote":"In addition, verifiers SHOULD perform an additional iteration of a keyed hashing or encryption operation using a secret key known only to the verifier.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"SP 800-63B-4 supersedes the earlier SP 800-63B, so it is the current revision.","quote":"This publication supersedes NIST Special Publication (SP) 800-63B.","verified":true,"document":"nist-sp800-63b-4"}]
judge unverified quotes
0
judge verdicts
[{"owasp-argon2id":{"status":"met","support":["E2"]},"owasp-conditions":{"status":"met","support":["E3"]},"nist-current":{"status":"met","support":["E6","E7","E9","E12"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E3","E4"]},"common-ground":{"status":"met","support":["E1","E6","E8","E9"]}},{"owasp-argon2id":{"status":"met","support":["E2"]},"owasp-conditions":{"status":"met","support":["E3"]},"nist-current":{"status":"met","support":["E6","E7","E9","E12"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E3","E4"]},"common-ground":{"status":"met","support":["E1","E6","E8","E9"]}},{"owasp-argon2id":{"status":"met","support":["E2"]},"owasp-conditions":{"status":"met","support":["E3"]},"nist-current":{"status":"met","support":["E6","E7","E9"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E3","E4"]},"common-ground":{"status":"met","support":["E1","E6","E8","E9"]}}]
judge failures
0
grounded weight share
0.875 (88%)

Final answer

Summary. The two sources are compatible. OWASP's default is Argon2id, and its FIPS-specific advice is PBKDF2. NIST SP 800-63B-4 does not name an algorithm in the text I read. A team that must meet FIPS should use PBKDF2-HMAC-SHA-256 at OWASP's iteration count (at least 600,000), with NIST's storage rules on top. I read only the OWASP cheat sheet and the NIST SP 800-63-4 web rendering of 63B-4. I did not open SP 800-132 or any FIPS 140 validation material, so those points are flagged below as unverified.

What OWASP recommends (https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html)

  • The general rule is that passwords "must be protected using strong, slow hashing algorithms such as Argon2id, bcrypt, or PBKDF2". Each password gets a unique salt.
  • The preference order is:

    • Argon2id first, with at least 19 MiB of memory, 2 iterations and parallelism 1.
    • scrypt if Argon2id is unavailable.
    • bcrypt only for legacy systems, with a work factor of 10 or more and a 72-byte password limit.
  • For FIPS it says: "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256."
  • The PBKDF2 section gives these iteration counts:

    • HMAC-SHA256: 600,000 (recommended).
    • HMAC-SHA512: 220,000.
    • HMAC-SHA1: 1,400,000, legacy only.
  • It suggests an optional pepper, a secret kept outside the database (for example in a secrets vault or HSM), as defense in depth.
  • It suggests keeping hashing under about one second, balancing security against login performance and denial-of-service risk.

What the current NIST SP 800-63B-4 requires (https://pages.nist.gov/800-63-4/sp800-63b.html)

  • SP 800-63B-4 is the current revision. Its text says it supersedes the earlier SP 800-63B.
  • The password-storage rules are:

    • Passwords SHALL be salted and hashed using a suitable password hashing scheme.
    • The salt SHALL be at least 32 bits and chosen to minimize collisions among stored hashes. The salt and the hash are both stored.
    • The cost factor SHOULD be as high as practical without hurting verifier performance, and SHOULD be increased over time.
    • A reference to the scheme and cost factor SHOULD be stored for each password, to allow migration.
    • "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used."
    • Verifiers SHOULD add an extra iteration of a keyed hash or encryption using a secret key known only to the verifier. The key comes from an approved random bit generator, is stored separately from the hashes, and should preferably be in an HSM or TEE.
  • The page does not name PBKDF2, Argon2, bcrypt or scrypt, and it gives no iteration count. The reference list cites SP 800-132 as "Recommendation for Password-Based Key…". I did not open SP 800-132, so I have not verified which scheme it approves.

Where the sources differ

  • OWASP says PBKDF2 "is permitted by NIST SP 800-63B-4". The 63B-4 text I read does not name PBKDF2. It only defers to SP 800-132 and future NIST guidance, so OWASP's claim rests on that indirect link.
  • NIST's approved-scheme language is a SHOULD. The SHALLs are salting, hashing, the 32-bit minimum salt, and storing the salt and hash.
  • The 600,000 figure is OWASP's, not NIST's.
  • OWASP ranks Argon2id first. NIST does not rank algorithms in this text.

How to reconcile them

  1. Use PBKDF2-HMAC-SHA-256 with at least 600,000 iterations for the FIPS-scoped service. This follows OWASP's FIPS exception and fits NIST's "approved scheme" language.
  2. Run it in a FIPS 140-validated cryptographic module. OWASP's stated reason for preferring PBKDF2 is that it "has FIPS-140 validated implementations". I did not check validation databases, so confirm that your specific module is validated.
  3. Follow NIST's storage rules. Use a unique random salt of at least 32 bits, and store the salt, the hash, and the scheme and cost factor in a versioned format.
  4. Add a secret-key step. Apply an HMAC with a key held in an HSM or secrets vault, separate from the database. This satisfies NIST's keyed-hash SHOULD and matches OWASP's pepper.
  5. Benchmark the iteration count so a hash takes under about a second and does not create a denial-of-service risk. Raise it over time and re-hash users at their next login.
  6. Do not rely on Argon2id alone for the FIPS-scoped path unless your assessor accepts it. This is my inference. The NIST text I read does not list Argon2id, and I did not verify its FIPS status. If the service has non-FIPS tiers, Argon2id with OWASP's parameters is the better choice there.
  7. Confirm with your compliance owner which obligations apply, and check the current SP 800-132 or newer NIST guidance. NIST 800-63B-4 covers federal digital identity, and FIPS 140 covers cryptographic module validation.
{"evidence": [
{"claim": "OWASP's general rule: passwords must be protected with strong, slow hashing algorithms (Argon2id, bcrypt, PBKDF2).", "quote": "Instead, they must be protected using strong, slow hashing algorithms such as Argon2id, bcrypt, or PBKDF2.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP's primary recommendation is Argon2id with minimum parameters of 19 MiB, 2 iterations and parallelism 1.", "quote": "Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "For FIPS-140 compliance OWASP recommends PBKDF2 with HMAC-SHA-256 and at least 600,000 iterations.", "quote": "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP says PBKDF2 should be preferred when FIPS-validated implementations are needed, and states that NIST SP 800-63B-4 permits it.", "quote": "Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "OWASP gives PBKDF2-HMAC-SHA256 at 600,000 iterations as the recommended setting.", "quote": "PBKDF2-HMAC-SHA256: 600,000 iterations (recommended)", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"},
{"claim": "NIST SP 800-63B-4 requires passwords to be salted and hashed with a suitable password hashing scheme.", "quote": "Passwords SHALL be salted and hashed using a suitable password hashing scheme.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST defers to approved schemes in SP 800-132 or updated NIST guidance, as a SHOULD, and does not name an algorithm in this text.", "quote": "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST requires a salt of at least 32 bits, with both salt and hash stored.", "quote": "The salt SHALL be at least 32 bits in length and chosen to minimize salt value collisions among stored hashes", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST says the cost factor should be as high as practical and increased over time.", "quote": "The chosen cost factor SHOULD be as high as practical without negatively impacting verifier performance. It SHOULD be increased over time to account for increases in computing performance.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST recommends storing a reference to the scheme and cost factor to allow migration.", "quote": "A reference to the password hashing scheme used, including the cost factor, SHOULD be stored for each password to allow migration to new algorithms and work factors.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "NIST recommends an additional keyed hash or encryption step using a secret key known only to the verifier.", "quote": "In addition, verifiers SHOULD perform an additional iteration of a keyed hashing or encryption operation using a secret key known only to the verifier.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"},
{"claim": "SP 800-63B-4 supersedes the earlier SP 800-63B, so it is the current revision.", "quote": "This publication supersedes NIST Special Publication (SP) 800-63B.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"}
]}

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.