The math behind the validation

On this page

How to read this page
1. Spearman’s rank correlation — ρ
2. Cohen’s quadratic-weighted kappa — κ
3. Planted-error detection rate
4. Precision, recall, and F₁ — FM3 scanner
5. Memorization index — contamination audit
6. Retry backoff — judge robustness

In plain language. The benchmark’s credibility rests on a handful of numbers: how well the AI judge agrees with a human expert, how reliably a scanner catches directional errors, and (soon) how much a model is reasoning versus reciting. This page defines each of those statistics exactly as the code computes them, says what counts as a good value, and reports where the project stands today. Nothing here is decorative — every formula maps to a function you can read in the repository.

How to read this page

Each statistic below is tagged with the validation step it serves and whether it is computed (a number exists today) or planned (the instrument is built, the run is pending). Symbols are defined immediately under each formula.

1. Spearman’s rank correlation — ρ

The code computes ρ as the Pearson correlation of the average ranks, which handles ties correctly:

\rho = \frac{\sum_{i=1}^{n} (R_i - \bar{R})(S_i - \bar{S})} {\sqrt{\sum_{i=1}^{n} (R_i - \bar{R})^2}\;\sqrt{\sum_{i=1}^{n} (S_i - \bar{S})^2}}

$n$ — number of jointly scored questions
$R_i$ — rank of the SME score on question $i$ (ties get the average rank)
$S_i$ — rank of the judge score on question $i$
$\bar{R}, \bar{S}$ — mean ranks

Reading guide: $\rho \ge 0.7$ rank order agrees · $0.4 \le \rho < 0.7$ partial · $\rho < 0.4$ disagreement on ordering.

Current values (n = 12): against the v0.2 judge prompt, core total $\rho = 0.770$ (passes); sign/directionality $\rho = 0.838$ (best-ranked dimension). Against the hardened v0.3.1 judge prompt (2026-07-09), core total $\rho = 0.421$ — the drop is driven entirely by the only two scores that moved, one of them the identity check firing on an answer the SME had accepted as a deliberate illustration (see §3 and the judge hardening writeup). At n = 12, both values are small-sample estimates.

2. Cohen’s quadratic-weighted kappa — κ

Define the disagreement weights, the observed agreement matrix, and the chance-expected matrix:

w_{ij} = \frac{(i - j)^2}{(K - 1)^2} \qquad E_{ij} = \frac{O_{i\bullet}\, O_{\bullet j}}{n}

\kappa = 1 - \frac{\sum_{i,j} w_{ij}\, O_{ij}}{\sum_{i,j} w_{ij}\, E_{ij}}

$K$ — number of score categories (for the core total, $K = 9$ , i.e. values $0\ldots 8$ )
$O_{ij}$ — observed count of questions the SME scored $i$ and the judge scored $j$
$O_{i\bullet}, O_{\bullet j}$ — row and column totals (marginals) of $O$
$E_{ij}$ — agreement expected if SME and judge scored independently
$w_{ij}$ — quadratic penalty for an $(i, j)$ disagreement; $0$ on the diagonal, $1$ at the extreme corners

Reading guide: $\kappa \ge 0.6$ substantial · $0.4 \le \kappa < 0.6$ moderate · $\kappa < 0.4$ weak.

Current values (n = 12): against the v0.2 judge prompt, core total $\kappa = 0.390$ (weak); sign/directionality $\kappa = 0.733$ (substantial); causal reasoning $\kappa = 0.000$ . Against the v0.3.1 prompt, core total $\kappa = 0.177$ — same two moved scores as the ρ drop above; the causal-reasoning $\kappa = 0$ persists unchanged.

3. Planted-error detection rate

For a single case, let $s^{\text{ctrl}}_{d}$ be the judge’s score on the clean reference and $s^{\text{def}}_{d}$ its score on the defective answer, both on the targeted dimension $d$ . Detection is the indicator:

\text{detected} = \mathbf{1}\!\left[\, s^{\text{def}}_{d} < s^{\text{ctrl}}_{d} \,\right]

The detection rate for a defect type $t$ (e.g. self_contradiction) over its set of cases $C_t$ is:

\text{rate}(t) = \frac{1}{|C_t|} \sum_{c \in C_t} \mathbf{1}\!\left[\, s^{\text{def}}_{d(c)} < s^{\text{ctrl}}_{d(c)} \,\right]

$d(c)$ — the dimension defect $c$ is designed to lower (a sign-flip → sign_directionality; a self-contradiction → causal_reasoning)
$\mathbf{1}[\cdot]$ — $1$ if the inequality holds, else $0$
$|C_t|$ — number of cases of type $t$ (3 per type in the original 21-case fixture; the cousin-product cell was expanded $3 \to 10$ for the v0.3 hardening, 28 cases total)

Current values (n = 28, v0.3.1 prompt, run 2026-07-09): all 18 non-cousin cases (sign flips, self-contradictions including subtle distributed ones, fabricated confidence, dropped edge cases) are caught at $1.00$ by both judges under the v0.2 prompt and stay near-perfect after hardening (two stable single-case Opus regressions, both caught by the second judge — panel detection 28/28). The historically weak cell was cousin-product mis-attribution (correct mechanics under a one-feature-off product name, e.g. autocall mechanics called a barrier reverse convertible): Opus $5/10$ , GPT-5.4 $8/10$ under the v0.2 prompt. The v0.3.1 judge prompt adds a mandatory product-identity check (both products named, an auditable product_identity_match flag, a deterministic factual-accuracy cap), raising cousin detection to Opus $9/10$ , GPT-5.4 $10/10$ . Original 21-case writeup: planted-error judge validation; hardening and 28-case re-run: judge hardening v0.3.

Why this beats a single external re-score for these dimensions

A sign-flip is a sign-flip independent of anyone’s judgment, so $\text{rate}(\texttt{sign\_flip})$ has no circularity — and it lands at $1.00$ . The judge also docks even subtle self-contradiction ( $1.00$ ), so the calibration’s $\kappa = 0$ is a subtlety gap the by-construction method cannot reach, not total blindness. The reproducible blind spot the set did surface — cousin-product mis-attribution, the FM1/FM2 confusion zone — is closed at the instrument level by the v0.3.1 identity check; the residual cost is that on real transcripts the check can over-fire on answers that use a neighbouring product as a deliberate illustration, which is what moved the small-sample ρ/κ in §1–§2.

4. Precision, recall, and F₁ — FM3 scanner

P = \frac{TP}{TP + FP} \qquad R = \frac{TP}{TP + FN} \qquad F_1 = \frac{2PR}{P + R} = \frac{2\,TP}{2\,TP + FP + FN}

$TP$ — scanner flagged a transcript the SME also marked as a directional error (true positive)
$FP$ — scanner flagged, SME did not (false positive)
$FN$ — SME marked an error the scanner missed (false negative)
$P$ (precision) — of the flags raised, the fraction that are real
$R$ (recall) — of the real errors, the fraction caught
$F_1$ — harmonic mean of $P$ and $R$

Current values (n = 12): $TP = 3,\ FP = 4,\ FN = 1,\ TN = 4 \Rightarrow P = 0.429,\ R = 0.750,\ F_1 = 0.545$ .

5. Memorization index — contamination audit

For an (original, twin) pair and a given model:

\text{MI} = \text{score}(\text{original}) - \text{score}(\text{twin})

A large positive MI means the model scored well on the canonical question but failed once a parameter moved — consistent with relying on memorized text.
An MI near zero means performance tracked the change — consistent with reasoning (necessary, not sufficient: a low MI is evidence of reasoning, not proof).

Aggregated per model and per domain, the distribution of MI flags any domain where the canonical answers may be contaminated by training data.

Current values (n = 12 twins, claude-sonnet-4-6, run 2026-06-13): mean MI $= +0.50$ /8 with 6/12 twins at exactly $0$ — no contamination signature. The residual positive signal concentrates in FM3 directional flips (skew $+5$ , issuer vega-sign $+4$ , correlation $+3$ ), and one negative MI ( $-4$ ) shows MI conflates contamination with intrinsic difficulty asymmetry — so the twin set doubles as an FM3-robustness probe, while the fictional-entity canary stays the cleaner pure-contamination test. Full writeup: docs/analysis/2026-06-13-contamination-audit.md.

6. Retry backoff — judge robustness

The delay before retry attempt $k$ (zero-indexed) is:

\Delta_k = 2^{k}\ \text{seconds} \quad\Rightarrow\quad 1\text{s},\ 2\text{s},\ 4\text{s},\ \ldots

After $\text{max\_retries}$ exhausted attempts the score is flagged errored = true and dropped from every aggregate — so it can never masquerade as a real zero in $\rho$ or $\kappa$ .