← all analysis · 2026-06-13 · docs/analysis/2026-06-13-contamination-audit.md

Contamination audit — perturbed-twin memorization index

Created: 2026-06-13 · Branch: validation-hardening Candidate: claude-sonnet-4-6 · Judge: claude-opus-4-6 · temperature 0.0 · n = 12 twin pairs Instrument: framework/perturbed_twins.md · runner: scripts/contamination_audit.py Parent: docs/superpowers/plans/2026-06-13-validation-hardening.md Step 4 · complements framework/canary_contamination_probe.md Tables are authored from a 2026-06-13 run; reproduce with python3 scripts/contamination_audit.py --model claude-sonnet-4-6 --judge claude-opus-4-6 --out /tmp/audit.md.

1. Why this artefact exists

The 56 benchmark questions derive from public SSPA/FINMA domain knowledge that is certainly in every frontier model’s training data, and the questionnaire itself sits in a public repository. The existing canary probe (framework/canary_contamination_probe.md) tests contamination with five fresh, fictional questions; the main 56 had no memorization-vs-reasoning check.

This audit adds one. For 12 of the main questions we built a perturbed twin: the same question with exactly one parameter inverted so the correct answer changes (skew steepens → flattens; barrier not breached → breached; correlation high → low; issuer short vega → long vega; …). A model that reasons tracks the inversion and scores well on the twin; a model that recites the canonical answer scores worse on the twin. The gap is the memorization index:

$\text{MI} = \text{core\_total}(\text{original}) - \text{core\_total}(\text{twin})$

Each side is judged against its own reference — the twin against its inverted reference — so a correct twin answer is not penalized for disagreeing with the canonical one.

2. Results

Aggregate (n = 12): mean original = 7.25/8, mean twin = 6.75/8, mean MI = +0.50. Six of twelve twins have MI = 0.

3. Findings

3.1 No evidence of gross contamination on this model. The mean MI is +0.50 on an 8-point scale, and half the twins show MI = 0 — the model scores the inverted question exactly as well as the canonical one. On D3 (continuous vs daily barrier), D4 (above- vs below-strike loss), D5-ADV (continuous-barrier breach), D7 (breach + below strike) and D12 (high- vs low-correlation coupon) the model tracked the inversion cleanly. If the benchmark were being answered from memorized text, we would expect systematically large positive MI; we do not see it.

3.2 The high-MI cases are all FM3 directional traps — and this is the more useful reading of the instrument. The three clear positive signals are skew direction (Q-D2-L3, +5), issuer vega sign on a capital-protection note (Q-D6-L1, +4), and worst-of correlation direction (Q-D1-L4, +3). These are exactly the sign/direction flips the project labels FM3. The model anchored to the canonical direction and lost points when the parameter moved. That can be either contamination or the inverted direction being intrinsically harder to reason about — the twin-MI cannot separate the two. So the honest interpretation is that the perturbed-twin MI is better understood as a directional-robustness / FM3-anchoring probe than as a pure contamination meter. For contamination specifically, the fictional-entity canary remains the cleaner instrument; for FM3 robustness, this twin set is the sharper one.

3.3 A negative MI confirms the conflation — and is itself informative. On Q-D11-L3 the model scored the canonical question lower (4/8) than the twin (8/8), MI = −4. The canonical question asks the genuinely counterintuitive direction (“worse issuer credit → higher coupon” via the funding-spread mechanism); the model fumbled it, while it handled the anomaly-flagging twin well. A negative MI is the opposite of a memorization signature — you cannot be reciting a canonical answer you score poorly on — and it demonstrates directly that MI measures (canonical difficulty − twin difficulty), not contamination alone.

3.4 Net. For claude-sonnet-4-6, the benchmark shows no contamination signature; the residual signal is concentrated in FM3 directional flips, which is consistent with the rest of the project’s findings (FM3 is the hard mode) rather than with recitation.

4. Limitations

• MI conflates contamination with intrinsic difficulty asymmetry (§3.2–3.3). A high MI flags a question worth inspecting; it is not by itself proof of training-data leakage. Read alongside the canary probe.
• One candidate model, 12 twins. This is a proof-of-method run (the chosen “smoke + proof” scope). Per-domain cells are n = 1–2; the aggregate is indicative, not a population estimate.
• Single run, temperature 0. No sampling-variance estimate; t = 0 is near-deterministic but not guaranteed.
• Author-written twins. The inverted reference answers are the author’s; a reviewer could contest an inversion. They are, however, mechanically checkable against standard SP definitions.

5. Implications and next steps

1. The benchmark is not obviously contaminated for this model — a defensible answer to “how do you know the models aren’t just reciting?” for claude-sonnet-4-6, complementing the fictional-entity canary.
2. Repurpose the twin set as an FM3-robustness probe across the model panel: run it on the frontier and open candidates (the 3-model or full-panel extension deferred from this run) and report the high-MI directional traps per model. This reuses the artefact for the project’s core FM3 question rather than only for contamination.
3. Keep the canary as the pure-contamination instrument; cite the two together in the methodology (mapping doc §6.1).
4. Expand twin coverage to the remaining domains (D9, D10, D13) and add ≥2 twins per domain before treating per-domain MI as anything more than anecdotal.