← all analysis · 2026-07-09 · docs/analysis/2026-07-09-judge-hardening-v03.md

Judge hardening v0.3: the product-identity check — design, validation, re-baseline

Author: Daniel Huber (executed via Claude Code) Date: 2026-07-09 Spec: docs/superpowers/specs/2026-07-07-judge-hardening-v03-design.md · Plan: docs/superpowers/plans/2026-07-07-judge-hardening-v03.md Companion artefacts: 2026-07-09-judge-v02-noise-floor.md · 2026-07-09-planted-v02-before.md · 2026-07-09-planted-v03-after.md · 2026-07-09-planted-v031-after.md · 2026-07-09-judge-v03-test-retest.md

TL;DR

The SP-Benchmark judge’s one confirmed blind spot — correct mechanics attributed to a cousin product one defining feature away — is closed at the instrument level. A mandatory Step 0 in the judge prompt now requires both judges to (1) name the product the reference concerns and the product the answer names/describes, (2) set an auditable product_identity_match boolean, which (3) deterministically caps factual_accuracy at 1 when false. On the by-construction planted set, cousin-cell detection rose from 5/10 (Opus) and 8/10 (GPT-5.4) to 9/10 and 10/10. The full 750-result corpus was re-judged twice under the hardened prompt (comparability break accepted by design, taken while the site is unpublished). Two honest costs are documented below: two stable single-case Opus regressions on non-cousin planted defects, and — on real transcripts — an identity-flag false positive against the SME’s judgement, which lowers the small-sample SME agreement stats rather than raising them.

1. The problem (confirmed 2026-06-13)

The planted-error study (2026-06-13-judge-validation-planted.md) found both panel judges detect sign flips, self-contradictions, fabrications, and dropped edge cases near-perfectly, but systematically miss cousin-product mis-attribution: an answer that keeps the reference’s mechanics verbatim while asserting a product identity one defining feature off (BRC vs plain reverse convertible, autocallable vs BRC, …). All shortfall in that study was this cell. The failed hypothesis of a “causal-reasoning ceiling” fix was retired then; the product-identity check was deferred (§7 of that writeup) and is delivered here.

2. The instrument (v0.3 → v0.3.1)

Schema (JudgeScore): product_identity_match: bool | None (None = pre-v0.3 result, never coerced to true), judge_prompt_version: str, and — added in the v0.3.1 iteration — reference_product / answer_product strings. When the judge sets product_identity_match: false, __post_init__ caps factual_accuracy at 1 in code, independent of whether the judge self-applied the rule; the raw judge JSON is preserved unmodified. Export CSV carries the new columns; legacy rows render blank.

Two prompt variants were tested:

• v0.3 — Step 0: “identify both products, compare defining features, set the flag.” Result: GPT-5.4 10/10 cousin detection; Opus unchanged within noise (6/10) — on every remaining miss it set pim=true, i.e. it compared mechanics to mechanics and never engaged with the stated product label.
• v0.3.1 (shipped) — Step 0 additionally requires filling reference_product and answer_product in the JSON and states that a correct-mechanics/wrong-name answer IS an identity mismatch. Forcing the judge to articulate both labels before scoring is what unlocked Opus. Prompt-version stamp: "0.3.1".

3. Validation chain

Detection ⟺ identity-flag coincide exactly on the cousin cell for both judges: the score drop is produced by the deterministic cap on an explicit, per-transcript, auditable flag — not by diffuse judge mood. That is the instrument working as designed.

Accepted trade-offs (Opus, stable across runs — all caught by GPT-5.4, panel 28/28)

• PE-26 (worst-of ↔ single-underlying): still missed — Opus does not treat the worst-of/correlation leg as identity-defining.
• PE-12 (fabricated SSPA code + product name): regression vs v0.2 — the identity framing makes Opus accept a confidently-stated label rather than challenge its existence.
• PE-14 (dropped edge case): single borderline case lost, likely attention displacement by the longer Step 0 preamble.

4. Effect on the real corpus

• Flag rates (primary judge Opus): 67/750 = 9%, of which 65 corroborated by GPT-5.4. GPT-5.4 alone flags a further 281 (46% total) — its threshold treats mechanics-level deviations as identity mismatches. Both judges’ flag rates rise monotonically as candidate models get weaker (Opus: 2% on claude-sonnet-4-6 → 16–25% on small local models; GPT: 25% → 68%), so the flags carry quality signal, but GPT-5.4’s flag should be read as a high-recall screen, Opus’s as the conservative signal — the same high-recall/low-precision division as the FM3 scanner (P 0.429 / R 0.750).
• Aggregate scores barely move: mean core_total on the 616 previously dual-judged results: Opus 5.85 → 5.91 (+0.07), GPT-5.4 4.69 → 5.03 (+0.34). The caps overwhelmingly land on answers already scoring ≤1 on factual accuracy; rankings are not upended by the re-baseline.
• Test-retest (750 results, two independent v0.3.1 runs): Opus ≥ 0.93 exact-match per dimension, pim stability 0.99; GPT-5.4 core_total exact 0.76 (mean |Δ| 0.27), pim stability 0.93. On real transcripts GPT-5.4 is the noisier instrument.

5. SME-calibration recompute — honest negative

Recomputing the 12-transcript SME comparison against the v0.3.1 primary-judge scores lowers the headline agreement: core-total Spearman ρ 0.770 → 0.421, quadratic-weighted κ 0.390 → 0.177. With n = 12 this is driven entirely by the only two scores that moved:

• Q-D5-L3-01: v0.3.1 Opus raised factual_accuracy 1 → 2 on an answer the SME scored 1 (internally contradictory) — a leniency flip away from the SME.
• Q-D5-ADV-01: the identity check fired (pim=false, “barrier option (generic, with knock-out example)” vs the BRC reference) on an answer the SME scored full marks, explicitly accepting the down-and-out illustration as legitimate. This is the identity check’s characteristic false positive: it cannot distinguish a wrong product claim from a deliberately generic illustrative example. GPT-5.4 concurs with the flag, so it is an instrument property, not judge idiosyncrasy.

Reading: the v0.3 instrument closes a by-construction blind spot the SME set never probed (none of the 12 real answers contained a cousin error), and pays for it with occasional over-flagging of legitimate illustration. For an MRM audience this is the right direction of error (conservative), but the claim “v0.3 improves SME agreement” is not supported — the causal-reasoning ceiling (judge mean 3.0, κ = 0) also persists unchanged. The external-SME ≥30-question re-score remains the calibration bar, and the 67 Opus-flagged transcripts are the natural rare-class set for the next human review pass.

6. Status and next steps

• v0.3.1 is live in src/sp_benchmark/judge.py (JUDGE_PROMPT_VERSION = "0.3.1"); all 750 stored results carry two v0.3.1 judge scores; the v0.2 corpus is archived locally (eval_results_v0.2_archive/, git-ignored; also recoverable from git history).
• Candidate v0.3.2 item (not started): tighten GPT-5.4’s flag threshold (“do not flag answers that use a generic or neighbouring product as an explicit illustration”) and re-run the planted set — must not lose the 10/10 cousin detection.
• Open: human review of the 67-flag set; external SME re-score (parked — outreach not currently possible); causal-reasoning ceiling (unaddressed by design in v0.3).