ScaffoldBench

RunHaiku 4.5repeat 2

Propose-select on OWASP vs NIST on password hashing

claude-haiku-4-5-20251001__propose-select__research-password-hashing__r2

Score

0.875

wrong answer
design ran as designed
Cost
$0.625
+ $0.067 judging
Latency
2m 38s
setup 0.0s, grading 4.7s
Turns
111
Tool calls
106
Subagents
3
Tokens in / out
2.2M / 29k
2,067,735 read from cache
Attempts
2
retried after API errors

Did the design run?

every check held
  • Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
  • No API failures: heldharness.apiNo model call failed at the API.
  • Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
  • No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
  • Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
  • No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
  • Builtin tools: heldharness.builtin-toolsA check recorded with the run.
  • Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
  • No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
  • Judge answered: heldharness.judgeAt least one grading call returned a usable verdict.
  • All three proposers ran: heldpropose-select.proposalsThree proposers ran in parallel, each from its own angle, and none was starved while another ran.
  • The selector chose: heldpropose-select.selectionThe selector picked a proposal by a valid number, read from its schema reply and not a default.
  • Each call was shown the right tools: heldpropose-select.toolsEach proposer was shown the read-only tools, the selector none, and the executor the full toolset.

Phases

Proposal 1 (policy-first): 0.0s to 2m 4s, finished, 30 tool calls, $0.171
Proposal 2 (state-first): 0.0s to 1m 56s, finished, 38 tool calls, $0.206
Proposal 3 (outcome-first): 0.0s to 1m 48s, finished, 38 tool calls, $0.206
Select: 2m 4s to 2m 38s, finished, 0 tool calls, $0.042

Drawn from the phase records: timing and cost per call, no tool calls or messages.

PhaseDurationCostTurnsToolsModel
Proposal 1 (policy-first)2m 4s$0.1713130claude-haiku-4-5-20251001
Proposal 2 (state-first)1m 56s$0.2063938claude-haiku-4-5-20251001
Proposal 3 (outcome-first)1m 48s$0.2063938claude-haiku-4-5-20251001
Select34.3s$0.04220claude-haiku-4-5-20251001

The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.

Runs you can replay

Record this cell yourself: npm run bench -- run --strategies propose-select --tasks research-password-hashing --models claude-haiku-4-5-20251001 --repeats 1

Score breakdown

  • passStates that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism.
    1.00weight 0.19

    met, backed by verified evidence E1. “Use Argon2id with a minimum configuration of 19 MiB of memory, 2 iterations, and 1 degree of parallelism.”

  • passStates OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required.
    1.00weight 0.13

    met, backed by verified evidence E2. “When FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more iterations and an internal hash function of HMAC-SHA-256.”

  • passStates that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used.
    1.00weight 0.19

    met, backed by verified evidence E4, E5. “An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.”

  • failStates that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording.
    0.00weight 0.13

    Not stated.

  • passExplains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply.
    1.00weight 0.25

    met, backed by verified evidence E2, E3. “**When FIPS-140 IS required**: Teams must use PBKDF2 with HMAC-SHA-256 and at least 600,000 iterations (per OWASP)”

  • passStates at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash.
    1.00weight 0.13

    met, backed by verified evidence E4, E1. “Both sources agree on the fundamentals: use salted hashing with appropriate cost/work factors that slow down password cracking attacks.”

Penalties

No penalties were applied.

Scorer metadata

corpus
{"version":2,"snapshot":"2026-10-01","hash":"e12dc326c69a32d91f85f6355dc63e2a7d672b605685a1ffa1582a42bb1c561e","documents":5}
opened documents
["owasp-password-storage","nist-sp800-63b-4","nist-sp800-63b-3"]
opened by role
{"primary":["owasp-password-storage","nist-sp800-63b-4"],"superseded":["nist-sp800-63b-3"]}
displayed share
{"owasp-password-storage":1,"owasp-authentication":0.0945,"rfc9106":0.1408,"nist-sp800-63b-4":0.3092,"nist-sp800-63b-3":0.1164}
cited count
3
verified count
3
hallucinated count
0
citation integrity
1
hallucinated urls
[]
evidence block
true
evidence block repaired
false
evidence count
7
verified quote count
6
verified quote rate
0.8571428571428571 (86%)
evidence
[{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP recommends Argon2id as the primary password hashing algorithm","quote":"Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP recommends PBKDF2 specifically for FIPS-140 compliance","quote":"If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP explicitly states PBKDF2 is preferred when FIPS-140 and NIST SP 800-63B-4 compliance are both required","quote":"Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.","verified":true,"document":"owasp-password-storage"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST SP 800-63B-4 requires passwords to be salted and hashed using a suitable password hashing scheme","quote":"Verifiers SHALL store passwords in a form that is resistant to offline attacks. Passwords SHALL be salted and hashed using a suitable password hashing scheme.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST SP 800-63B-4 references approved password hashing schemes from SP 800-132 or updated NIST guidelines","quote":"An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST SP 800-63B-4 requires minimum 32-bit salt and increasing cost factors over time","quote":"The salt SHALL be at least 32 bits in length and chosen to minimize salt value collisions among stored hashes (i.e., to prevent multiple subscriber accounts from having the same hashed password). The chosen cost factor SHOULD be as high as practical without negatively impacting verifier performance.\n…[truncated 84 chars]","verified":false,"document":"nist-sp800-63b-4","failure":"the quote does not occur in the page"},{"url":"https://pages.nist.gov/800-63-3/sp800-63b.html","claim":"NIST SP 800-63B Revision 3 recommended PBKDF2 and memory-hard functions","quote":"Examples of suitable key derivation functions include Password-based Key Derivation Function 2 (PBKDF2) [SP 800-132] and Balloon [BALLOON]. A memory-hard function SHOULD be used because it increases the cost of an attack.","verified":true,"document":"nist-sp800-63b-3"}]
judge unverified quotes
0
judge verdicts
[{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2"]},"nist-current":{"status":"met","support":["E4","E5","E6"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E4","E6","E1"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2"]},"nist-current":{"status":"met","support":["E4","E5","E6"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E4","E6","E2"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2"]},"nist-current":{"status":"met","support":["E4","E5","E6"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E1","E2","E4","E6"]}}]
judge failures
0
grounded weight share
0.875 (88%)

Final answer

Complete Answer

OWASP Password Storage Guidance

OWASP's Password Storage Cheat Sheet provides the following recommendations for password hashing algorithms:

  1. Primary recommendation (Argon2id): Use Argon2id with a minimum configuration of 19 MiB of memory, 2 iterations, and 1 degree of parallelism.
  2. Secondary recommendation (scrypt): If Argon2id is unavailable, use scrypt with CPU/memory cost parameter 2^17, block size of 8 (1024 bytes), and parallelization parameter of 1.
  3. Legacy systems (bcrypt): For legacy systems, use bcrypt with a work factor of 10 or more and password limit of 72 bytes.
  4. FIPS-140 compliance (PBKDF2): When FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more iterations and an internal hash function of HMAC-SHA-256.

OWASP explicitly states: "Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required."

NIST SP 800-63B-4 Requirements

The current revision (800-63B-4) provides less prescriptive but still concrete guidance. NIST SP 800-63B Revision 3 was superseded by NIST SP 800-63B-4 as of August 1, 2025.

Verifiers "SHALL store passwords in a form that is resistant to offline attacks. Passwords SHALL be salted and hashed using a suitable password hashing scheme." The requirement further states: "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used."

NIST SP 800-63B-4 specifies:

  • Salt must be at least 32 bits in length and chosen to minimize collisions
  • Cost factor should be as high as practical without negatively impacting verifier performance
  • Cost factor should be increased over time to account for computing performance improvements
  • Verifiers SHOULD perform an additional iteration using a secret key (pepper) stored separately from hashed passwords

The Reconciliation: OWASP and NIST with FIPS-140

The apparent tension between OWASP's preference for Argon2 and FIPS-140 compliance is resolved by algorithm choice based on regulatory requirements:

When FIPS-140 is NOT required: Teams should follow OWASP's primary recommendations—Argon2id is the best choice because it is memory-hard and more resistant to GPU and ASIC attacks than PBKDF2.

When FIPS-140 IS required: Teams must use PBKDF2 with HMAC-SHA-256 and at least 600,000 iterations (per OWASP) because:

  • Argon2 does not have FIPS-140 validated implementations
  • PBKDF2 is specified in NIST SP 800-132 (referenced by NIST SP 800-63B-4)
  • PBKDF2 has widely available FIPS-140 validated implementations
  • PBKDF2 meets the NIST SP 800-63B-4 requirement for using "an approved password hashing scheme"

Both sources agree on the fundamentals: use salted hashing with appropriate cost/work factors that slow down password cracking attacks. The key difference is simply which algorithms can be formally validated to FIPS-140 standards.


{"evidence": [
  {
    "claim": "OWASP recommends Argon2id as the primary password hashing algorithm",
    "quote": "Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.",
    "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"
  },
  {
    "claim": "OWASP recommends PBKDF2 specifically for FIPS-140 compliance",
    "quote": "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.",
    "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"
  },
  {
    "claim": "OWASP explicitly states PBKDF2 is preferred when FIPS-140 and NIST SP 800-63B-4 compliance are both required",
    "quote": "Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.",
    "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"
  },
  {
    "claim": "NIST SP 800-63B-4 requires passwords to be salted and hashed using a suitable password hashing scheme",
    "quote": "Verifiers SHALL store passwords in a form that is resistant to offline attacks. Passwords SHALL be salted and hashed using a suitable password hashing scheme.",
    "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"
  },
  {
    "claim": "NIST SP 800-63B-4 references approved password hashing schemes from SP 800-132 or updated NIST guidelines",
    "quote": "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.",
    "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"
  },
  {
    "claim": "NIST SP 800-63B-4 requires minimum 32-bit salt and increasing cost factors over time",
    "quote": "The salt SHALL be at least 32 bits in length and chosen to minimize salt value collisions among stored hashes (i.e., to prevent multiple subscriber accounts from having the same hashed password). The chosen cost factor SHOULD be as high as practical without negatively impacting verifier performance. It SHOULD be increased over time to account for increases in computing performance.",
    "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"
  },
  {
    "claim": "NIST SP 800-63B Revision 3 recommended PBKDF2 and memory-hard functions",
    "quote": "Examples of suitable key derivation functions include Password-based Key Derivation Function 2 (PBKDF2) [SP 800-132] and Balloon [BALLOON]. A memory-hard function SHOULD be used because it increases the cost of an attack.",
    "url": "https://pages.nist.gov/800-63-3/sp800-63b.html"
  }
]}

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.