ScaffoldBench

RunHaiku 4.5repeat 1

Gated-execute on OWASP vs NIST on password hashing

claude-haiku-4-5-20251001__gated-execute__research-password-hashing__r1

Score

0.875

wrong answer
design ran as designed
Cost
$0.207
+ $0.068 judging
Latency
1m 33s
setup 0.0s, grading 4.9s
Turns
39
Tool calls
38
Subagents
0
Tokens in / out
813k / 8.3k
766,754 read from cache

Did the design run?

every check held
  • Run completed: heldharness.run-completedThe environment was built, the architecture returned and grading finished, none of them throwing.
  • No API failures: heldharness.apiNo model call failed at the API.
  • Every call ended: heldharness.calls-endedEvery call ended with a result, or was stopped by the run's own limits.
  • No harness tool denied: heldharness.harness-toolsThe sandbox guard never denied a harness tool, nor a tool the session itself offered.
  • Tool servers connected: heldharness.mcp-connectedEvery tool server the harness gave a call was connected when the session started.
  • No foreign tool servers: heldharness.mcp-hermeticNo call loaded a tool server the harness did not give it.
  • Builtin tools: heldharness.builtin-toolsA check recorded with the run.
  • Structured replies arrived: heldharness.structured-outputEvery call that asked for structured output and ran to the end got it.
  • No silent fallbacks: heldharness.not-degradedNo phase fell back on a default (an empty plan, a default winner) instead of its output.
  • Judge answered: heldharness.judgeAt least one grading call returned a usable verdict.
  • Every state change was reviewed: heldgated-execute.reviewedEvery call that changed the world had been allowed by a review before it ran, counted from the world’s own journal.
  • Every decision was read: heldgated-execute.decisionsEvery review that ran to its end gave its decision through the schema; an unreadable one is a denial.
  • Each call was shown the right tools: heldgated-execute.toolsThe executor was shown the task’s full toolset and each reviewer the read-only tools.

Phases

Execute: 0.0s to 1m 33s, finished, 38 tool calls, $0.207

Drawn from the phase records: timing and cost per call, no tool calls or messages.

PhaseDurationCostTurnsToolsModel
Execute1m 33s$0.2073938claude-haiku-4-5-20251001

The call-by-call replay is not published for this run. Traces are large, so the repository keeps 9 of this run set’s 900 and leaves the rest out; the score, phases and final answer on this page come from the committed run record.

Runs you can replay

Record this cell yourself: npm run bench -- run --strategies gated-execute --tasks research-password-hashing --models claude-haiku-4-5-20251001 --repeats 1

Score breakdown

  • passStates that OWASP recommends Argon2id first, with a minimum configuration of 19 MiB of memory, an iteration count of 2 and 1 degree of parallelism.
    1.00weight 0.19

    met, backed by verified evidence E1. “Primary choice**: Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism”

  • passStates OWASP's alternatives as conditions: scrypt if Argon2id is not available, bcrypt only for legacy systems, and PBKDF2 (at least 600,000 iterations with HMAC-SHA-256) if FIPS-140 compliance is required.
    1.00weight 0.13

    met, backed by verified evidence E2. “Legacy systems only**: bcrypt with a work factor of 10 or more”

  • passStates that the current NIST SP 800-63B (revision 4) requires passwords to be salted and hashed with a suitable password hashing scheme that takes a cost factor, and says an approved scheme from the latest revision of SP 800-132 (or updated NIST guidelines) SHOULD be used.
    1.00weight 0.19

    met, backed by verified evidence E4, E5. “"Passwords SHALL be salted and hashed using a suitable password hashing scheme" and "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used."”

  • failStates that the superseded revision 3 named PBKDF2 and Balloon as example key derivation functions and said a memory-hard function SHOULD be used, and that revision 4 no longer contains that wording.
    0.00weight 0.13

    Not stated.

  • passExplains that for this team the two do not actually conflict: OWASP's own guidance is to use PBKDF2 (HMAC-SHA-256, at least 600,000 iterations) when FIPS-140 compliance is required, and NIST permits PBKDF2, so PBKDF2-HMAC-SHA-256 satisfies both; Argon2id remains OWASP's choice only where FIPS does not apply.
    1.00weight 0.25

    met, backed by verified evidence E2, E3, E5. “Use PBKDF2 with HMAC-SHA-256 and 600,000 iterations as OWASP specifies. PBKDF2 is an "approved password hashing scheme" under NIST SP 800-63B-4”

  • passStates at least one requirement both sources share, such as a unique salt for every password or a deliberately expensive hash with a tunable cost (work) factor instead of a fast general-purpose hash.
    1.00weight 0.13

    met, backed by verified evidence E4. “Both sources agree that the work factor or cost parameter should be adjusted over time”

Penalties

No penalties were applied.

Scorer metadata

corpus
{"version":2,"snapshot":"2026-10-01","hash":"e12dc326c69a32d91f85f6355dc63e2a7d672b605685a1ffa1582a42bb1c561e","documents":5}
opened documents
["owasp-password-storage","nist-sp800-63b-4"]
opened by role
{"primary":["owasp-password-storage","nist-sp800-63b-4"]}
displayed share
{"owasp-password-storage":1,"owasp-authentication":0.0485,"rfc9106":0.0673,"nist-sp800-63b-4":0.187,"nist-sp800-63b-3":0.013}
cited count
2
verified count
2
hallucinated count
0
citation integrity
1
hallucinated urls
[]
evidence block
true
evidence block repaired
false
evidence count
6
verified quote count
6
verified quote rate
1
evidence
[{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP recommends Argon2id as the primary password hashing algorithm with specific parameters","quote":"Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP recommends PBKDF2 for FIPS-140 compliance","quote":"If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.","verified":true,"document":"owasp-password-storage"},{"url":"https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html","claim":"OWASP states PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations","quote":"Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.","verified":true,"document":"owasp-password-storage"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST SP 800-63B-4 requires passwords to be salted and hashed using a suitable password hashing scheme","quote":"Passwords SHALL be salted and hashed using a suitable password hashing scheme. Password hashing schemes take a password, a salt, and a cost factor as inputs and generate a password hash.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST SP 800-63B-4 allows either FIPS-approved or NIST-recommended password hashing schemes","quote":"An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.","verified":true,"document":"nist-sp800-63b-4"},{"url":"https://pages.nist.gov/800-63-4/sp800-63b.html","claim":"NIST defines approved cryptography as either FIPS-approved or NIST-recommended","quote":"An encryption algorithm, hash function, random bit generator, or similar technique that is Federal Information Processing Standards (FIPS)-approved or NIST-recommended. Approved algorithms and techniques are either specified or adopted in a FIPS or NIST recommendation.","verified":true,"document":"nist-sp800-63b-4"}]
judge unverified quotes
0
judge verdicts
[{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"partial","support":["E2"]},"nist-current":{"status":"met","support":["E4","E5"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E2","E3","E5"]},"common-ground":{"status":"met","support":["E4"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2"]},"nist-current":{"status":"met","support":["E4","E5"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E2","E3"]},"common-ground":{"status":"met","support":["E4"]}},{"owasp-argon2id":{"status":"met","support":["E1"]},"owasp-conditions":{"status":"met","support":["E2"]},"nist-current":{"status":"met","support":["E4","E5"]},"nist-revision":{"status":"missing","support":[]},"reconciliation":{"status":"met","support":["E2","E3","E5"]},"common-ground":{"status":"met","support":["E4"]}}]
judge failures
0
grounded weight share
0.875 (88%)

Final answer

Now I have all the information I need to provide a comprehensive answer. Let me compile the findings.

Answer

A team must reconcile two different but complementary guidance sources when selecting a password hashing algorithm for FIPS compliance:

OWASP Password Storage Guidance Recommendations:

The OWASP Password Storage Cheat Sheet provides a clear hierarchy of password hashing algorithms. It states: "Passwords should never be stored in plain text. Instead, they must be protected using strong, slow hashing algorithms such as Argon2id, bcrypt, or PBKDF2." More specifically, OWASP recommends:

  • Primary choice: Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism
  • Secondary choice: scrypt with a minimum CPU/memory cost parameter of 2^17
  • Legacy systems only: bcrypt with a work factor of 10 or more
  • For FIPS-140 compliance: "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256."

The OWASP cheat sheet further explains that "Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required."

NIST SP 800-63B-4 Requirements:

The current revision of NIST SP 800-63B-4 (Revision 4) does not mandate specific password hashing algorithms by name. Instead, it requires: "Passwords SHALL be salted and hashed using a suitable password hashing scheme" and "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used."

NIST SP 800-63B-4 defines "approved cryptography" as: "An encryption algorithm, hash function, random bit generator, or similar technique that is Federal Information Processing Standards (FIPS)-approved or NIST-recommended. Approved algorithms and techniques are either specified or adopted in a FIPS or NIST recommendation."

This definition is important because it allows for two categories of approved algorithms:

  1. FIPS-approved algorithms (such as PBKDF2, which is specified in NIST SP 800-132)
  2. NIST-recommended algorithms (which could include modern algorithms like Argon2)

How to Reconcile the Two:

The two sources align by recognizing that password hashing requirements depend on compliance constraints:

  1. For systems without FIPS requirements: Follow OWASP's preference hierarchy (Argon2id > scrypt > bcrypt), as these are modern, memory-hard algorithms that resist GPU and ASIC attacks while remaining efficient for legitimate use. These align with NIST's flexibility in accepting any "approved" (NIST-recommended) password hashing scheme.
  2. For systems with FIPS-140 compliance requirements: Use PBKDF2 with HMAC-SHA-256 and 600,000 iterations as OWASP specifies. PBKDF2 is an "approved password hashing scheme" under NIST SP 800-63B-4 because it is specified in NIST SP 800-132, making it FIPS-approved and satisfying both federal compliance requirements and NIST's guidance.
  3. Work factors and cost: Both sources agree that the work factor or cost parameter should be adjusted over time to account for increases in computing performance, without negatively impacting verifier performance.

The apparent difference between OWASP's modern algorithm preferences (Argon2id, scrypt) and NIST's reference to SP 800-132 (which primarily covers PBKDF2) is not actually a conflict. NIST SP 800-63B-4 explicitly allows "updated NIST guidelines on password hashing schemes," suggesting openness to newer algorithms. OWASP's conditional recommendation of PBKDF2 for FIPS compliance reflects the practical reality that only PBKDF2 currently has both FIPS-140 validated implementations and the necessary specification status to satisfy federal requirements.


{"evidence": [
  {
    "claim": "OWASP recommends Argon2id as the primary password hashing algorithm with specific parameters",
    "quote": "Use Argon2id with a minimum configuration of 19 MiB of memory, an iteration count of 2, and 1 degree of parallelism.",
    "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"
  },
  {
    "claim": "OWASP recommends PBKDF2 for FIPS-140 compliance",
    "quote": "If FIPS-140 compliance is required, use PBKDF2 with a work factor of 600,000 or more and set with an internal hash function of HMAC-SHA-256.",
    "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"
  },
  {
    "claim": "OWASP states PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations",
    "quote": "Since PBKDF2 is permitted by NIST SP 800-63B-4 and has FIPS-140 validated implementations, it should be the preferred algorithm when these are required.",
    "url": "https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html"
  },
  {
    "claim": "NIST SP 800-63B-4 requires passwords to be salted and hashed using a suitable password hashing scheme",
    "quote": "Passwords SHALL be salted and hashed using a suitable password hashing scheme. Password hashing schemes take a password, a salt, and a cost factor as inputs and generate a password hash.",
    "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"
  },
  {
    "claim": "NIST SP 800-63B-4 allows either FIPS-approved or NIST-recommended password hashing schemes",
    "quote": "An approved password hashing scheme published in the latest revision of [SP800-132] or updated NIST guidelines on password hashing schemes SHOULD be used.",
    "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"
  },
  {
    "claim": "NIST defines approved cryptography as either FIPS-approved or NIST-recommended",
    "quote": "An encryption algorithm, hash function, random bit generator, or similar technique that is Federal Information Processing Standards (FIPS)-approved or NIST-recommended. Approved algorithms and techniques are either specified or adopted in a FIPS or NIST recommendation.",
    "url": "https://pages.nist.gov/800-63-4/sp800-63b.html"
  }
]}

Provenance of the numbers on this page

run set
scaffold-or-scale-v1
models
Haiku 4.5, Sonnet 5.5, Opus 5.5
effort
high
repeats
2
fixtures
replay
runs
900 / 900
strategy cost
$354.44
judge cost
$9.19
wall clock
9h 48m
git
fce5807b2d

Cost excludes judge calls, which are tracked separately so a scoring model never flatters or penalises an architecture. Every figure on this site is read from results/ at build time.