ScaffoldBench

RunHaiku 4.5repeat 1

Propose-select on OWASP vs NIST on password hashing

claude-haiku-4-5-20251001__propose-select__research-password-hashing__r1

Score

0.578

wrong answer
design ran as designed
Cost
$0.568
+ $0.085 judging
Latency
2m 37s
setup 0.0s, grading 5.0s
Turns
108
Tool calls
103
Subagents
3
Tokens in / out
1.7M / 29k
1,534,052 read from cache

Did the design run?

every check held
  • Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
  • No API failures: heldharness.apiNo model call failed at the API.
  • Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
  • No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
  • Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
  • No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
  • Builtin tools: heldharness.builtin-toolsA check recorded with the run.
  • Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
  • No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
  • Judge answered: heldharness.judgeAt least one grading call returned a usable verdict.
  • All three proposers ran: heldpropose-select.proposalsThree proposers ran in parallel, each from its own angle, and none was starved while another ran.
  • The selector chose: heldpropose-select.selectionThe selector picked a proposal by a valid number, read from its schema reply and not a default.
  • Each call was shown the right tools: heldpropose-select.toolsEach proposer was shown the read-only tools, the selector none, and the executor the full toolset.

Phases

Proposal 1 (policy-first): 0.0s to 1m 25s, finished, 36 tool calls, $0.182
Proposal 2 (state-first): 0.0s to 1m 11s, finished, 27 tool calls, $0.152
Proposal 3 (outcome-first): 0.0s to 1m 23s, finished, 40 tool calls, $0.178
Select: 1m 25s to 2m 37s, finished, 0 tool calls, $0.056

Drawn from the phase records: timing and cost per call, no tool calls or messages.

PhaseDurationCostTurnsToolsModel
Proposal 1 (policy-first)1m 25s$0.1823736claude-haiku-4-5-20251001
Proposal 2 (state-first)1m 11s$0.1522827claude-haiku-4-5-20251001
Proposal 3 (outcome-first)1m 23s$0.1784140claude-haiku-4-5-20251001
Select1m 12s$0.05620claude-haiku-4-5-20251001

The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.

Runs you can replay

Record this cell yourself: npm run bench -- run --strategies propose-select --tasks research-password-hashing --models claude-haiku-4-5-20251001 --repeats 1

Score breakdown

  • passStates that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism.
    1.00weight 0.19

    met, backed by verified evidence E1. “The preferred recommendation is Argon2id with minimum configuration of 19 MiB of memory, 2 iterations, and 1 degree of parallelism.”

  • partialStates OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required.
    0.50weight 0.13

    partial, backed by verified evidence E2. “When Argon2id is not available, OWASP recommends scrypt with cost parameter of 2^17, block size of 8, and parallelization parameter of 1.”

  • passStates that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used.
    1.00weight 0.19

    met, backed by verified evidence E4, E5. “NIST SP 800-63B-4 requires that "Passwords SHALL be salted and hashed using a suitable password hashing scheme."”

  • failStates that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording.
    0.00weight 0.13

    Not stated.

  • partialExplains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply.
    0.50weight 0.25

    partial, backed by verified evidence E2, E3. “Teams must use PBKDF2 with HMAC-SHA-256 at a minimum of 600,000 iterations.”

  • partialStates at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash.
    0.13weight 0.13

    partial, but no evidence entry supports it. “they must be protected using strong, slow hashing algorithms such as Argon2id, bcrypt, or PBKDF2.”

Penalties

No penalties were applied.

Scorer metadata

corpus
{"version":2,"snapshot":"2026-10-01","hash":"e12dc326c69a32d91f85f6355dc63e2a7d672b605685a1ffa1582a42bb1c561e","documents":5}
opened documents
["owasp-password-storage","nist-sp800-63b-4","nist-sp800-63b-3"]
opened by role
{"primary":["owasp-password-storage","nist-sp800-63b-4"],"superseded":["nist-sp800-63b-3"]}
displayed share
{"owasp-password-storage":1,"owasp-authentication":0.089,"rfc9106":0.1177,"nist-sp800-63b-4":0.2237,"nist-sp800-63b-3":0.0709}
cited count
2
verified count
2
hallucinated count
0
citation integrity
1
hallucinated urls
[]
evidence block
true
evidence block repaired
false
evidence count
8
verified quote count
8
verified quote rate
1
evidence
[{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP recommends Argon2id as the strongest algorithm for password hashing in general use","quote":"Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP specifies PBKDF2 for FIPS-140 compliance","quote":"If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP explains that PBKDF2 should be preferred when FIPS compliance is required because it has FIPS-validated implementations","quote":"Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.","verified":true,"document":"owasp-password-storage"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST SP 800-63B-4 requires passwords to be salted and hashed using a suitable password hashing scheme","quote":"Passwords SHALL be salted and hashed using a suitable password hashing scheme.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST SP 800-63B-4 references approved password hashing schemes from SP800-132 or updated NIST guidelines","quote":"An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST defines approved cryptography as FIPS-approved or NIST-recommended algorithms","quote":"approved cryptography: An encryption algorithm, hash function, random bit generator, or similar technique that is Federal Information Processing Standards (FIPS)-approved or NIST-recommended. Approved algorithms and techniques are either specified or adopted in a FIPS or NIST recommendation.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST SP 800-63B-4 distinguishes between approved algorithms and FIPS140-validated implementations for general users","quote":"Authenticators used at AAL1 SHALL use approved cryptography. In other words, they must use approved algorithms, but the implementation need not be validated under [FIPS140].","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST requires FIPS140 Level 1 validation for verifiers operated by federal agencies","quote":"Cryptography used by verifiers operated by or on behalf of federal agencies at AAL1 SHALL be validated to meet the requirements of [FIPS140] Level 1.","verified":true,"document":"nist-sp800-63b-4"}]
judge unverified quotes
0
judge verdicts
[{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"partial","support":["E2"]},"nist-current":{"status":"met","support":["E4","E5"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"partial","support":[]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"partial","support":["E2"]},"nist-current":{"status":"met","support":["E4","E5"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"partial","support":["E2","E3"]},"common-ground":{"status":"partial","support":[]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"partial","support":["E2"]},"nist-current":{"status":"met","support":["E4","E5"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"partial","support":["E2","E3"]},"common-ground":{"status":"partial","support":["E4"]}}]
judge failures
0
grounded weight share
0.75 (75%)

Final answer

Analysis and Recommendation

A team choosing a password hashing algorithm for a web service with US federal (FIPS) requirements should understand that OWASP and NIST differ on their primary recommendations, but these differences can be reconciled by prioritizing FIPS compliance when required.

What OWASP Recommends

OWASP's Password Storage Cheat Sheet provides a clear hierarchy of algorithms based on security effectiveness. The guidance states: "Passwords should never be stored in plain text. Instead, they must be protected using strong, slow hashing algorithms such as Argon2id, bcrypt, or PBKDF2." The preferred recommendation is Argon2id with minimum configuration of 19 MiB of memory, 2 iterations, and 1 degree of parallelism. When Argon2id is not available, OWASP recommends scrypt with cost parameter of 2^17, block size of 8, and parallelization parameter of 1. However, critically, OWASP acknowledges the FIPS requirement: "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256."

Additionally, OWASP explains the rationale for PBKDF2 when FIPS compliance is needed, noting: "Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required."

What NIST SP 800-63B-4 (Current Revision) Requires

NIST SP 800-63B-4 requires that "Passwords SHALL be salted and hashed using a suitable password hashing scheme." Regarding which schemes to use, NIST states that "an approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used." This language indicates NIST references both the 2010 SP800-132 standard (which covers PBKDF2) and allows for updated guidance.

Importantly, NIST SP 800-63B-4 distinguishes between "approved cryptography" (which can be either FIPS-approved or NIST-recommended algorithms) and the requirement for FIPS 140 Level 1 validation. For non-federal users, NIST permits NIST-recommended algorithms: "Authenticators used at AAL1 SHALL use approved cryptography. In other words, they must use approved algorithms, but the implementation need not be validated under [FIPS140]."

However, for federal agencies specifically, a stricter requirement applies: "Cryptography used by verifiers operated by or on behalf of federal agencies at AAL1 SHALL be validated to meet the requirements of [FIPS140] Level 1." This means teams operating verifiers for federal agencies cannot use Argon2id, scrypt, or bcrypt—even though NIST accepts these as approved algorithms in general.

How to Reconcile the Two Sources

The apparent conflict dissolves when examining FIPS requirements carefully:

  1. For Non-FIPS Systems: Teams should follow OWASP's preferred recommendations and use Argon2id as the first choice, since it offers superior security against both GPU and memory-based attacks. Argon2id is an "approved algorithm" under NIST SP 800-63B-4 and qualifies for general use.
  2. For FIPS-140 Compliance (Federal Agencies): Teams must use PBKDF2 with HMAC-SHA-256 at a minimum of 600,000 iterations. This is because PBKDF2 is built using FIPS-approved cryptographic primitives (HMAC and SHA-256), while Argon2, scrypt, and bcrypt are not FIPS-approved password hashing algorithms despite their superior security characteristics.
  3. The Strategic Path: If a federal system needs both strong security and FIPS compliance, the recommendation is:

    • Use PBKDF2 with HMAC-SHA-256 and 600,000+ iterations initially to meet FIPS 140 requirements
    • Monitor NIST's "updated guidelines on password hashing schemes" mentioned in SP 800-63B-4, as NIST may eventually approve Argon2 or similar algorithms
    • Plan for future migration to stronger algorithms as FIPS approvals evolve
    • Ensure iteration counts increase over time as computing power increases

The key insight is that OWASP's conditional guidance acknowledges this exact constraint: when FIPS-140 compliance is required, the choice is predetermined to PBKDF2, despite Argon2id being objectively more secure in other contexts. NIST's definition of "approved cryptography" is broader (including NIST-recommended algorithms), but FIPS 140 validation narrows the acceptable choices for federal agencies.


{"evidence": [{"claim": "OWASP recommends Argon2id as the strongest algorithm for password hashing in general use", "quote": "Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"}, {"claim": "OWASP specifies PBKDF2 for FIPS-140 compliance", "quote": "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"}, {"claim": "OWASP explains that PBKDF2 should be preferred when FIPS compliance is required because it has FIPS-validated implementations", "quote": "Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.", "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"}, {"claim": "NIST SP 800-63B-4 requires passwords to be salted and hashed using a suitable password hashing scheme", "quote": "Passwords SHALL be salted and hashed using a suitable password hashing scheme.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"}, {"claim": "NIST SP 800-63B-4 references approved password hashing schemes from SP800-132 or updated NIST guidelines", "quote": "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"}, {"claim": "NIST defines approved cryptography as FIPS-approved or NIST-recommended algorithms", "quote": "approved cryptography: An encryption algorithm, hash function, random bit generator, or similar technique that is Federal Information Processing Standards (FIPS)-approved or NIST-recommended. Approved algorithms and techniques are either specified or adopted in a FIPS or NIST recommendation.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"}, {"claim": "NIST SP 800-63B-4 distinguishes between approved algorithms and FIPS140-validated implementations for general users", "quote": "Authenticators used at AAL1 SHALL use approved cryptography. In other words, they must use approved algorithms, but the implementation need not be validated under [FIPS140].", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"}, {"claim": "NIST requires FIPS140 Level 1 validation for verifiers operated by federal agencies", "quote": "Cryptography used by verifiers operated by or on behalf of federal agencies at AAL1 SHALL be validated to meet the requirements of [FIPS140] Level 1.", "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"}]}

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.