← all analysis · · docs/analysis/2026-06-13-judge-validation-planted.md
Judge validation — planted-error set
On this page
Created: 2026-06-13 · Branch:
validation-hardeningInstrument:framework/judge_validation_planted_errors.md(21 cases) · runner:scripts/judge_validation_planted.pyParent:docs/superpowers/plans/2026-06-13-validation-hardening.mdStep 2 · companion to2026-05-sme-calibration.mdTables below are authored (not auto-overwritten); regenerate the raw tables withpython3 scripts/judge_validation_planted.py --judge claude-opus-4-6,gpt-5.4 --out /tmp/planted.md.
1. Why this artefact exists
The SME calibration (2026-05-sme-calibration.md) validates the judge against a single human rater (the project author) on n = 12 — useful, but circular: the same person wrote the questions, the reference answers, and the scoresheet. The sharpest finding there was a causal-reasoning ceiling: the judge awarded 3/3 on every question, including four chains the SME flagged as internally contradictory, collapsing quadratic-weighted κ to 0.000 on that dimension.
This artefact removes the circularity for the dimensions where it can. Each case takes a correct reference answer from the real questionnaire and injects exactly one known defect. Because a sign-flip is a sign-flip by construction, the ground truth is not a matter of judgment — anyone can verify it — so there is no rater and no circularity. We then ask: when the judge sees the defective answer, does it dock the scoring dimension that defect targets, relative to the clean reference (control)?
This is a different instrument from framework/canary_contamination_probe.md (which tests whether the candidate model memorizes). Here the judge is under test.
2. Method
21 cases. Each defect maps by construction to the dimension that must drop:
| Defect type | Injected error | Targeted dimension | n |
|---|---|---|---|
sign_flip | a directional claim reversed | sign_directionality | 3 |
self_contradiction | a causal chain that contradicts itself (3 stark + 3 subtle/distributed) | causal_reasoning | 6 |
wrong_attribution | a correct mechanism attributed to the wrong product (severity gradient) | factual_accuracy | 6 |
fabricated_confidence | a non-existent SSPA code / FINMA circular asserted as real | factual_accuracy | 3 |
dropped_edge_case | the edge case the reference flags silently omitted | edge_case_awareness | 3 |
For each case the judge scores the clean reference (control) and the defective variant. Detection is score_defective < score_control on the targeted dimension. Run at temperature 0.0 against the two production judges (Claude Opus 4.6, GPT-5.4). No case errored; every control scored full marks on its targeted dimension (sign 2/2, causal 3/3, factual 2/2, edge 1/1), so no “miss” is an artefact of the control itself scoring low.
The set was expanded from 15 to 21 cases on 2026-06-13 (Step 2.9, “hold prompt, expand set”): +3 subtle self-contradictions (PE-16–18: distributed/numeric/implicit rather than adjacent X-then-not-X) to probe where causal detection breaks down, and +3 wrong_attribution cases (PE-19–21) spanning a mislabel-severity gradient. The judge prompt was deliberately not changed, preserving comparability with the v0.2 calibration.
3. Results
Detection rate by defect type:
| Defect type | Targeted dim | n | Opus 4.6 | GPT-5.4 |
|---|---|---|---|---|
| sign_flip | sign_directionality | 3 | 1.00 (3/3) | 1.00 (3/3) |
| self_contradiction | causal_reasoning | 6 | 1.00 (6/6) | 1.00 (6/6) |
| wrong_attribution | factual_accuracy | 6 | 0.50 (3/6) | 0.67 (4/6) |
| fabricated_confidence | factual_accuracy | 3 | 1.00 (3/3) | 1.00 (3/3) |
| dropped_edge_case | edge_case_awareness | 3 | 1.00 (3/3) | 1.00 (3/3) |
| all | — | 21 | 0.86 (18/21) | 0.90 (19/21) |
wrong_attribution detail (the one weak dimension), ordered by mislabel severity:
| Case | Mislabel | Severity | Opus | GPT-5.4 |
|---|---|---|---|---|
| PE-07 | BRC decomposition → “tracker certificate” | structurally distinct | ✅ | ✅ |
| PE-19 | BRC replication → “capital-protected note” | opposite risk profile | ✅ | ✅ |
| PE-21 | autocallable → “bonus certificate” | structurally distinct | ✅ | ✅ |
| PE-09 | barrier-gamma cost → “plain vanilla European call” | one-feature cousin (barrier) | ❌ | ✅ |
| PE-08 | autocall mechanics → “barrier reverse convertible” | one-feature cousin (autocall) | ❌ | ❌ |
| PE-20 | BRC decomposition → “reverse convertible” | one-feature cousin (barrier) | ❌ | ❌ |
self_contradiction detail (stark vs subtle):
| Case | Kind | Opus ctrl→def | GPT ctrl→def |
|---|---|---|---|
| PE-04/05/06 | stark (adjacent X / not-X) | 3→0/1/2 ✅ | 3→1/0/2 ✅ |
| PE-16 | subtle — numeric (1.2% vs stated 1.5% floor) | 3→1 ✅ | 3→2 ✅ |
| PE-17 | subtle — principle violated late (“no free lunch” → “no cost”) | 3→2 ✅ | 3→2 ✅ |
| PE-18 | subtle — concentration framed as diversification | 3→1 ✅ | 3→1 ✅ |
4. Findings
4.1 The judge detects internal contradiction — even subtle, distributed ones. All six self-contradictions, including the three deliberately subtle cases (numeric threshold violation, principle reversed several sentences later, concentration recast as diversification), were docked by both judges. The planted method therefore could not reproduce the calibration’s κ = 0 ceiling: there is no by-construction self-contradiction in this set blatant enough to be unarguable yet subtle enough to fool the judge. Two readings remain open, and the planted method cannot separate them:
- the four undetected calibration chains were subtler still than PE-16–18, or
- they sit at the boundary between the SME’s contradiction threshold and an objective one (the SME called them contradictory; a neutral reader, and the judge, might not). Resolving this needs the four actual calibration answers re-examined — which is not by-construction ground truth — so it is a genuine limit of this instrument, recorded as such.
Note the contrast paradigm matters: the judge scores the contradictory answer low in absolute terms here (causal 0–2), whereas in calibration it awarded 3/3 to answers scored standalone. A clean reference sitting alongside may be what surfaces the defect. This suggests a methodological lever for the benchmark itself (reference-anchored scoring), noted for future work.
4.2 The genuine, reproducible blind spot is cousin-product mis-attribution. With six attribution cases the pattern is sharp and survives both judges:
- Structurally-distinct mislabels are caught (PE-07 tracker, PE-19 capital-protected note, PE-21 bonus certificate — all ✅✅). When the named product has an obviously different risk profile or building blocks, the judge docks factual accuracy.
- One-feature “cousin” mislabels are missed (PE-08 autocall↔BRC, PE-20 RC↔BRC, PE-09 vanilla↔barrier). When the named product differs from the correct one by a single defining feature (a barrier, an autocall trigger), the judge rewards the correct mechanics and overlooks the wrong label. PE-08 and PE-20 are missed by both judges; PE-09 by Opus only. This is the FM1/FM2 zone made precise: a model can describe the right plumbing under a nearly-right product name and keep full factual-accuracy credit. It is exactly the confusion most likely to occur in practice (RC vs BRC, autocall vs BRC are daily confusions), which makes it the most consequential judge weakness found.
4.3 Confident fabrication is caught; correct-but-mislabelled is not. Invented SSPA codes / FINMA circulars were docked 3/3 (factual 2→1). The discriminator is whether the asserted content is itself false (caught) or true-but-misassigned (missed when the misassignment is to a cousin product).
4.4 Aggregate. Opus 4.6 detects 18/21 (0.86), GPT-5.4 19/21 (0.90). The entire shortfall is concentrated in cousin-product attribution; every other defect type is detected at 1.00 by both judges.
5. Limitations
- By-construction blatancy bounds the claim. High detection means “the judge catches the constructed version.” For self_contradiction this is now a strong floor (subtle cases caught); for the calibration ceiling it means the instrument cannot reach the failure — §4.1.
- n is small per cell (3–6). The attribution gradient (§4.2) establishes the existence and shape of the cousin-product blind spot, not its precise prevalence.
- Single run, temperature 0. No sampling-variance estimate.
- Author-chosen defects. The defects are by-construction verifiable; the selection of defect types is the author’s.
6. Implications for the methodology
- Directional, causal, edge-case, and fabrication scoring are now externally defensible without a human SME — all detect 1.00 across both judges by construction. For FM3 specifically (sign_directionality) this is stronger evidence than the n = 12 single-rater calibration.
- The judge’s one material weakness is cousin-product mis-attribution. This is now documented with a reproducible gradient. The fix is a future judge-prompt product-identity check (deferred — see §7), but the benchmark can already report the blind spot honestly.
- A reference-anchored scoring variant is worth exploring (§4.1): the judge docks defects far more reliably when a clean reference sits alongside the answer than when scoring standalone.
- Feeds the open-call protocol (Step 5). “Self-calibrated, with a by-construction judge-validation set anyone can reproduce and extend, and an honestly-reported judge blind spot” is the citable framing this artefact underwrites.
7. Decision taken (Step 2.9)
Chosen route: hold the prompt, expand the set (preserve v0.2 comparability; map the failure before touching the production prompt). Executed above — the expansion located the failure precisely (cousin-product attribution) and established that the causal ceiling is not reproducible by construction. A targeted product-identity prompt fix and its before/after remain available as future work, to be run as a deliberate, comparability-breaking change once the benchmark is ready to re-baseline.
CLOSED 2026-07-09. The deferred prompt fix shipped as judge v0.3.1: the fixture grew to 28 cases (cousin cell 3 → 10, PE-22…PE-28), and the deliberate comparability break was taken — full corpus re-judged twice under the hardened prompt. Cousin-cell detection: Opus 5/10 → 9/10, GPT-5.4 8/10 → 10/10, with an auditable per-transcript
product_identity_matchflag and deterministic factual-accuracy cap. Before/after, trade-offs (two stable Opus single-case regressions), and an honest SME-recompute negative are in /analysis/2026-07-09-judge-hardening-v03/.