Explainability: worked examples

In plain language. When an AI model produces an answer, can we trace why it produced that specific answer (not just from the input, but through the chain of internal computation that led there)? The question matters for finance because regulators and model risk managers need to evaluate whether an AI’s reasoning is sound, not just whether its output happens to be right. This page reports two early attempts at producing that kind of trace, on small open-weight models. Neither is a production-ready audit tool; both are experiments showing what’s technically possible today and where the limits sit.

Methodologically these are applied-interpretability experiments: CLT attribution graphs and a combined NLA/SAE readout on open-weight models (Gemma-2-2B, Gemma-3-12B-IT; GemmaScope-2 SAE at residual-stream layer 32). They stand or fall on interpretability grounds first: whether a technique faithfully recovers a model’s internal computation, and where it breaks.

In plain terms: an attribution graph traces which internal features drove a specific output; a sparse autoencoder (SAE) is a tool that decomposes a model’s internal state into human-nameable features; a natural-language autoencoder (NLA) makes the model describe its own internal state in words; and the dual readout runs the last two side by side so each can check the other.

The FINMA framing below is one application of that: the regulatory case for why per-decision faithfulness matters in a Swiss bank. It is not the reason the methods are interesting. Negative results are reported as findings in their own right, not despite a compliance motive.

This page reports work on per-decision causal documentation for LLMs, anchored where relevant to FINMA Guidance 08/2024 focus area VI (“Explainability”). Two complementary techniques are reported here: cross-layer transcoder (CLT) attribution graphs on small base models, and a per-token dual readout combining natural-language autoencoders (NLA) with sparse autoencoder (SAE) features on a production-class instruct model.

First worked example: barrier knock-in on Gemma-2-2B (May 2026)

A first attribution graph has been computed on the canonical barrier- knock-in prompt using circuit-tracer. The full writeup covers methodology, per-input-position attribution, top contributing features, FINMA §VI mapping, and the CLT method’s limitations.

Headline finding. On the prompt “In a barrier reverse convertible, if the underlying closes below the barrier at maturity, the investor”, base Gemma-2-2B’s top-1 next-token prediction is " receives" (p = 0.346), not " loses" (rank 8, p = 0.015). The base 2B model’s surface behaviour is formulaic, not domain-aware. The attribution graph nevertheless reveals that barrier-related input tokens do contribute to the " loses" logit; the causal-chain signal exists internally but is dominated by template-completion computation at the final token. The artefact demonstrates that per-decision causal documentation is technically feasible today at the small-scale end of the model spectrum.

Read the full analysis: 2026-05: First attribution graph. Top-features tables, reproduction scripts, and prompt sets are part of the research record.

Contrastive variant: does the barrier direction flip the circuit? (May 2026)

The single-prompt worked example left one question open: even though the output is wrong, do the model’s internal features still encode the causal chain? A contrastive run answers it: flip the one word that flips the economics (below → above) and check whether the circuit flips with it. Full writeup of the contrastive run.

Headline finding (a negative result, and that is the point). Flipping the underlying from below the barrier to above it (an economically opposite scenario) barely moves the model. Both prompts predict " receives" as top-1 at p ≈ 0.347 (a 0.002 difference); the top-15 logit-driver features are the same circuit (10 of 11 identical, same #1 feature at L19); and 8–9 of the top-10 barrier-concept features are shared. Gemma-2-2B (base) has barrier-token features but no representation of the barrier’s directional consequence that attribution can recover.

This corrects the cautious earlier reading: the “98 concept-position features” seen for the " loses" logit are not specific to the loss scenario; substantially the same features fire for the gain scenario. For an MRM reviewer the value is methodological: a contrastive design can falsify a faithfulness claim, and an explainability method that cannot fail such a test is not an audit. Faithfulness must be re-tested per model.

Read the full analysis: 2026-05-16: Contrastive attribution. Per-prompt top-features tables are in the research record.

Per-token dual readout on Gemma-3-12B-IT (May 2026)

The Gemma-2-2B contrastive run flagged its own faithfulness limit explicitly: a base 2B model is not the surface a Swiss MRM reviewer deploys. The natural next test is a production-class instruct model, Gemma-3-12B-IT, with a method that goes beyond logit-attribution to a two-channel readout of the model’s internal state at the decision layer.

The technique is a per-token dual readout at residual-stream layer 32:

• a natural-language autoencoder (NLA) verbalization (Anthropic 2026; checkpoints at kitft/nla-models) producing a free-text explanation for each answer token, and
• a GemmaScope-2 sparse autoencoder decomposition (DeepMind, March 2026; google/gemma-scope-2-12b-it) producing a discrete feature signature for the same token.

Reading both channels together yields a candidate vocabulary of four agreement patterns (converge, nla_only, sae_only, confabulation) proposed as method-confidence tags on a model’s output. Whether they are useful to an MRM reviewer is exactly what the at-scale run below tests, and the answer so far is “not yet, on this model.” See the headline distribution and the caveats.

Proof of protocol on the barrier triple

The proof-of-protocol run applied the dual readout to the same below/above/neutral barrier triple used in the Gemma-2-2B contrastive test, this time on Gemma-3-12B-IT (47 tokens × 3 prompts). The two channels independently identified financial-domain onset at the same token positions, and the SAE/NLA cross-check caught the run’s first NLA confabulation: a verbal claim not corroborated by any SAE feature at that position. Full writeup with the per-regime token table.

At scale: 25 SP-Benchmark questions

The at-scale run applied the same protocol to a 25-question subset of the SP-Benchmark questionnaire spanning 7 domains (Structuring, Pricing & Greeks, Lifecycle Events, Risk Transfer, Maturity & Settlement, SSPA Taxonomy, Credit-Linked / Fixed Income). The subset was selected by behavioural-signal ranking across 14 closed-frontier and open-weight models in the existing eval panel. Each question’s per-token NLA verbalizations and SAE feature signatures were captured to JSON, then tagged by the agreement-pattern taxonomy. Full at-scale writeup, covering the per-question table, three FM-specific case studies, and aggregates.

Headline distribution (25 questions).

By domain and by failure mode. The same 25 questions decomposed two ways: across the 7 SP-Benchmark domains the subset spans, and across the three failure modes (FM1 surface, FM2 isolated, FM3 directional) the questions target. A question with multiple FM targets contributes to each FM row.

Worked case study: Q-D7-L3-01, European vs American barrier semantics. The closest in-the-wild parallel to the proof-of-protocol barrier-direction contrast. The question asks an MRM-relevant compare: a BRC with a European barrier (checked only at maturity) vs an otherwise identical BRC with an American barrier (continuous, breached 3 months ago, recovered). The model produces a fluent answer that inverts the BRC coupon condition and gives a self-contradictory European-barrier definition. The dual readout reads as follows:

• NLA themes (aggregate, answer span): structured product definition · barrier/knock-out mechanics · European vs American comparison · conditional trigger logic. The verbal channel names the correct concept space but never produces a contradicting claim; themes co- occur with the wrong answer rather than flagging it.
• Top SAE features (TF-IDF on L32 16k/l0_big): #16295 on the token barrier is domain-relevant; the other two top features fire on generic tokens (at, any). One discriminating feature, two wrapper.
• Tag: confabulation (auto-tagged by Claude Opus 4.6, human-confirmed). Fires on both the BRC coupon inversion and the self-contradictory European-barrier definition.

MRM implication. The dual readout catches the error only because the auto-tagger has access to the reference answer. Without the reference, the readout produces concept-relevant themes that look corroborating. This bounds what the planned FINMA mapping can claim: on this model+SAE pair, for FM3-class directional errors, the dual readout supports post-hoc audit (reference-anchored) but not pre-hoc flagging. The proof-of-protocol claim (“the readout can flag a directional flip”) narrows into the at-scale in-the-wild finding (“the readout does not flag a directional flip on an uncontrolled prompt without external ground-truth”).

Methodology caveats

Two caveats are load-bearing for any external citation of the at-scale numbers:

• SAE feature-degeneracy at L32 16k/l0_big. Naive frequency or magnitude ranking of SAE features produces near-identical signatures across all 25 questions: feature #15019 peaks at ~22,000 on every question, #313 fires on 94% of answer tokens regardless of topic. The tabulation pipeline uses a TF-IDF score (peak activation × log(1 + N/n_present)) with #15019 hard-excluded as a documented generic. Even with this mitigation, ~6 of 25 questions produce only reasoning-wrapper features; for those questions converge and sae_only are unavailable as tags. The protocol is partially capacity-limited at this layer/SAE. The follow-up has since been run: an A/B against the 16×-wider 262k SAE at the same layer (2026-06-13) shows partial de-degeneration — an instrument upgrade, not a rescue — written up as Appendix A of the at-scale analysis.

• Taxonomy v1.2: what confabulation does and does not mean. The v1.2 clarification (2026-05-27) reserves confabulation for NLA verbalizing a claim contradicted by the prompt or by other model state, not for NLA aligning with a model answer that happens to disagree with the reference. The dual readout speaks to method confidence, not model correctness. A question can receive a converge tag and still be factually wrong; correctness is the judge’s job, not the interpretability layer’s.

Inter-rater note

The 9 confabulation auto-calls in the at-scale run received one independent human review. 7/9 agreed (78%); two were downgraded to nla_only because they were concept hallucinations rather than token-level contradictions. This is a single-rater pass on the highest-stakes subset, not the full subject-matter-expert calibration the testing-validation page §6.2 still flags as an open methodological gap.

Read the full analyses.

• NLA sidecar (final-token readout, infrastructure proof)
• Per-token dual readout on the barrier triple (proof of protocol)
• At scale: 25 SP-Benchmark questions (tag distribution, case studies, aggregates)
• Agreement-pattern taxonomy (frozen v1)
• Taxonomy v1.2 clarification (what the dual-readout certifies)

Per-question JSONs, the tabulation script, and the execution plan are internal research artefacts not yet published.

Next steps

• Alternative SAE width — done (2026-06-13), partial result. The 262k-width A/B on the six SAE-blank questions shows the L32 degeneracy was substantially an instrument artefact of the narrow 16k dictionary (single-feature dominance roughly halves, distinct features roughly double) — but not entirely: a new generic feature still tops 4 of 6 questions, so the wider dictionary surfaces the domain signal without yet producing clean question-distinct signatures. Details in Appendix A of the at-scale analysis. The remaining variant worth testing is a different layer.
• Independent-rater calibration for the dual-readout tags. A single-rater review of the highest-stakes subset (the confabulation auto-calls) is done — see the inter-rater note above — but the full independent-rater pass on the tag taxonomy remains open, as the load-bearing external-validity check on the auto-tagger.
• FINMA reference-number mapping (conditional). A formal evaluator → rule → FINMA mapping table only makes sense if the SAE channel improves beyond the partial 262k result above. Writing it on top of the current readout would dress a negative result as a method. If the balance does not improve, the deliverable is a write-up of why two-channel cross-validation fails at this layer. (Also gated on sourcing the “four cross-validation rules” and “13 langfuse-finma evaluators” referenced in the research plan.)
• GemmaScope feature interpretation lookup for named feature IDs in the per-token readout tables via Neuronpedia auto-interp labels. (For the 262k run this is limited: Neuronpedia hosts only the 16k source for the 12B, so 262k features have no hosted auto-interp labels.)