← all analysis · · docs/analysis/2026-05-agreement-patterns-taxonomy.md

Agreement-Pattern Taxonomy v1 (Path A Week 3 tagging frame)

Date: 2026-05-24 Status: Frozen for Week 3 tagging. Revisions during Week 3 must increment to v1.1 and re-tag from the start. Anchors: /analysis/2026-05-22-dual-readout-week2/ §6 (proto-rules table). Used by:

• mechinterp/scripts/06_run_w3_batch.py (forthcoming) — populates agreement_pattern field per question.
• /analysis/2026-05-dual-readout-empirical/ (forthcoming) — tabulates results per pattern.
• Path A Week 4 — formalizes each tag → FINMA Randziffer (deferred to docs/superpowers/plans/2026-05-track-b-path-a-six-week-execution.md §Week 4).

Why a frozen taxonomy

Per the Week 3 prework memo (patha-week3-prework task 2): “Lock the taxonomy before tagging starts, otherwise vocabulary drifts mid-run.” Each of the 25 Week 3 questions gets exactly one agreement_pattern tag based on the dominant cross-method behaviour observed during the model’s answer-generation tokens. Without a frozen frame, the tags from question 1 and question 25 would not be comparable.

Scope of a tag

One tag per question. The tag describes the dominant cross-method behaviour across the answer-generation token positions (typically ~10–30 tokens after the prompt boundary). If multiple patterns fire equally at different positions, use mixed and enumerate in the notes field rather than forcing a single bucket.

The four patterns

1. converge — NLA + SAE independently identify the same concept

Definition: NLA verbalization and SAE top-K features at answer-relevant token positions both point to the same finance-domain concept, independently arrived at.

Tagging rule:

• NLA explanations at ≥2 answer-relevant token positions name a coherent finance concept (e.g. specific product class, payoff mechanic, market-event language), AND
• SAE top-10 at those positions contains features from the dominant structured-product cluster identified in Week 2 (#15019 and conceptual neighbourhood) OR a coherent SP-related feature set discriminable from baseline.

Week 2 anchor: Positions 5–18 across all three barrier prompts. Both methods independently identified structured-product content from position 5 ( convertible) onward; feature #15019 co-occurred with NLA “financial/investment” framing.

Confidence claim: High — independent methods converge. Strongest readout we can produce with the current protocol.

2. nla_only — NLA verbalizes a claim the SAE does not corroborate

Definition: NLA produces a specific finance claim (a direction, a magnitude, a product mechanic) at a token position where the SAE top-K does not visibly discriminate or support that claim.

Tagging rule:

• NLA verbalization at the answer-critical position names a specific direction, magnitude, or payoff, AND
• SAE top-10 at the same position is dominated by generic finance-context features (the same features that fire at contrasting questions / contrasting prompts), AND
• No SAE feature in the top-10 is identifiably specific to the claim being made.

Week 2 anchor: Final-token ( investor, pos 18) on Prompt 1: NLA names “loss of principal”; SAE top-3 (#15019, #608, #313) is byte-identical to Prompt 2’s “equivocal” verbalization. The directional claim is suggestive but not corroborated by the top-K SAE leg.

Confidence claim: Hypothesis — treat as tentative, not actionable on its own.

3. sae_only — SAE discriminates where NLA does not

Definition: SAE top-K shows a discriminating feature (different active features for two semantically distinct conditions, e.g. below vs above, FM3-positive vs FM3-negative) at a position where the NLA verbalization does not call out the corresponding concept.

Tagging rule:

• SAE top-10 differs meaningfully between this question and a comparable contrast (same domain, same token sequence up to the contrast point) at a specific token position, AND
• NLA verbalization at that position does not mention the discriminating concept in plain language.

Week 2 anchor: Position 11 of barrier prompts — SAE feature #15663 distinguishes below from above (#20); neither NLA verbalization at pos 11 explicitly mentions direction. The SAE sees a signal the actor’s English does not surface.

Confidence claim: Unverbalized signal — possible blind spot in verbal readout. Flag for follow-up: feature-level auto-interp lookup (Neuronpedia) may name the latent concept the NLA is missing.

4. confabulation — NLA names a fact contradicted by context

Definition: NLA verbalization at any answer-relevant position contains a factual claim that contradicts (a) the prompt text at or before that position, or (b) the model’s eventual answer, or (c) the reference answer for that question.

Tagging rule:

• NLA explanation at any answer-relevant position contains a domain claim (product class, direction, mechanic), AND
• That claim is directly contradicted by an earlier prompt token, the model’s final answer, or the SP-Benchmark reference answer.

Week 2 anchor:

• Pos 12 of Prompt 1 — NLA says “underlying asset is above” when the prompt token at pos 11 is below (within-prompt contradiction).
• Every prompt’s NLA mis-names the product (“ETF/bond hybrid”, “Enhanced Yield Note”, “callable convertible bond”) despite the prompt stating “barrier reverse convertible” (reference-answer contradiction).

Confidence claim: Downgrade — the NLA leg should not be trusted at this position. Distinct from nla_only (which is unsupported but not actively contradicted).

Tagging procedure

1. Read the NLA explanations across the question’s answer-generation tokens.
2. Read the SAE top-10 features at the same positions.
3. Compare against the question’s reference answer (and, where applicable, a contrasting question — see §“Contrast pairing” below).
4. Apply the four rules in this priority order: confabulation first (highest-cost error to miss), then sae_only, then nla_only, then converge as default if none of the others fire.
5. If multiple patterns fire at roughly equal weight across different positions, set agreement_pattern = "mixed" and enumerate per-position calls in notes.

The priority order encodes a cost asymmetry: a missed confabulation is worse than a missed converge. When in doubt, tag the more conservative pattern.

Contrast pairing

The Week 2 anchors all depend on having a contrasting prompt (below vs above) to detect sae_only (feature discrimination across the contrast) and to characterize converge (same features fire across both → not direction-specific). The SP-Benchmark Week 3 questions are not naturally paired, so:

• For questions within the same domain at adjacent difficulty levels (L1 vs L2, L2 vs L3): treat the adjacent-difficulty question as the contrast.
• For directional questions (e.g. D2 Pricing & Greeks, D7 Maturity & Settlement): if the answer flips sign in a corresponding question, pair them.
• For questions with no obvious contrast: tag against the baseline behaviour observed at out-of-domain (early-position) tokens from §4.1 of the Week 2 doc. sae_only is harder to justify without a contrast — note that limitation when it fires.

Schema

Each Week 3 question JSON contains:

{
  "question_id": "Q-D2-L3-01",
  "agreement_pattern": "converge | nla_only | sae_only | confabulation | mixed",
  "notes": "Free-text. If pattern=mixed, enumerate per-position calls here."
}

Confidence-claim → FINMA Randziffer (deferred to Week 4)

Each pattern maps one-step to a confidence level that the Week 4 work will bind to FINMA 2017/1 Randziffer (model-output plausibility, §VI). The taxonomy is designed so this mapping is mechanical, not a re-interpretation:

What this taxonomy does not claim

• It is not a behavioural taxonomy of model errors. It is a cross-method-agreement taxonomy. A question can have a converge tag and still be wrong (both methods agree the model is in the structured-product context, but the model still misclassifies the SSPA code). The agreement pattern speaks to method confidence, not to model correctness.
• It does not require quantitative SAE-feature attribution. Top-K=10 is a coarse readout; a missed sae_only (because the discriminating feature is at rank 15) is acceptable false-negative noise at this resolution.
• It is not stable across SAE generations or NLA checkpoints. The frozen v1 is bound to GemmaScope 2 16k/l0_big at L32 of Gemma-3-12B-IT, and NLA kitft/nla-gemma3-12b-L32-av. If either changes, re-version.

v1.2 addendum (2026-05-27)

A v1.1 auto-tagger pass and its diagnostic (docs/analysis/2026-05-taxonomy-v1.1-delta.md) surfaced a load-bearing ambiguity in rule 4 that v1.1’s “concept-hallucination vs token-contradiction” refinement did not actually resolve: clause (c) of rule 4 (“contradiction with the reference answer”) silently conflicts with §“What this taxonomy does not claim,” which already commits the taxonomy to method-confidence reporting rather than correctness validation.

The clarification is written up at /analysis/2026-05-taxonomy-v1-2-clarification/. Headline:

• v1.2 = drop clause (c) from rule 4. The dual-readout discriminates whether NLA verbalizes what the model’s internal state says — it does not certify the model’s answer against ground truth. Confabulation fires only on contradictions against (a) the prompt or (b) the model’s own answer.
• Converge-on-error subcase is now explicit. If the model is wrong and NLA reinforces the wrong answer, that is converge (if SAE corroborates) or nla_only (if SAE is silent), not confabulation.
• Four W3 canonical tags shift under v1.2 (Q-D1-L1-01, Q-D7-L1-01, Q-D7-L3-01, Q-D12-L3-01). They are annotated in §4.1 of 2026-05-dual-readout-empirical.md rather than re-tagged; the published §4.2/§4.3/§6 aggregates remain on the v1.0 frame.
• Cost-saving: Option B (read the v1.1 auto_reasoning fields directly) established that running v1.2 against the auto-tagger would only flip Q-D6-L4-01 and Q-D8-L3-01 — the four over-corrections cannot move because v1.1 already reached them via clause (a)/(b) reasoning. No third auto-tagger pass was run.

For new Path B / SP-Benchmark expansion work, use the v1.2 rule 4 text from the clarification doc, not the v1 rule 4 body above. The v1 body is frozen for W3 traceability only.