← all analysis · · docs/analysis/2026-05-first-attribution-graph.md

First Attribution Graph — Barrier Knock-in on Gemma-2-2B

Date: 2026-05-05 Model: google/gemma-2-2b (base, not instruction-tuned, bf16) Tool: circuit-tracer 0.5.x with cross-layer transcoders from GemmaScope (mwhanna/gemma-scope-transcoders, all 26 layers) Hardware: GB10 (Grace Blackwell, 16 GB resident model+transcoder footprint) Prompt: In a barrier reverse convertible, if the underlying closes below the barrier at maturity, the investor

Run artefacts: mechinterp/outputs/barrier_knockin_v1/

• top_features.md — full per-logit attribution tables (sections 2–4 of this doc draw from it)
• run.log — circuit-tracer CLI stdout
• run_metadata.json — provenance
• graph_files/graph-metadata.json — visualizer metadata (large barrier_knockin_v1.json and .pt are gitignored; regenerate via mechinterp/scripts/03_attribution_graph.py)

1. Research question

Does Gemma-2-2B internally represent the causal link from “underlying closes below the barrier” to “investor loses principal protection” as an identifiable feature path through its layers, or is the next-token continuation it produces driven by surface co-occurrence and template-matching with no domain-aware computation?

For an MRM reader: this is the minimum viable explainability artefact. If a 2B base model with no finance fine-tuning shows nothing of the causal chain, that is the floor; if it shows something, the same machinery applied to a production-class model can in principle yield per-decision causal documentation.

2. Method

Tool. circuit-tracer builds an attribution graph by replacing each transformer layer with a pretrained cross-layer transcoder (CLT) — a sparse autoencoder factoring the residual stream into discrete “features.” Edges in the graph are gradient-based attributions: how much node j contributes to the output of node i.

Run. One forward pass on the 19-token prompt, attribution computed against the top-10 next-token logits (cumulative probability 0.81). 20,193 features fired across the 26 layers; circuit-tracer kept the 7,500 highest-attribution features, plus 494 “error” nodes (one per layer × position, capturing residual not represented by transcoders), 19 token-embedding nodes, and 10 logit nodes — adjacency matrix 8023 × 8023, ~17 M nonzeros.

Spot-check criterion. From the underlying plan: “if the graph is empty or dominated by a single trivial feature, the target-token position or CLT layer is wrong.” Graph has 2,712 used nodes and 1.0 M used edges across 26 layers — non-trivial; passes.

What we report. (a) The model’s own predicted next tokens with probabilities. (b) Per-input-position aggregate attribution — sum of |attribution| from all features+errors+embeds at each input position to a chosen output logit. (c) Top-15 source nodes per logit. (d) Features specifically at concept-of-interest positions (barrier, below, underlying, closes, maturity).

3. Result

3.1 The model’s surface behaviour is formulaic, not domain-aware

The top-1 next-token prediction is receives (p = 0.346). The expected canonical continuation, loses, ranks 8th at p = 0.015 — 23× less likely than receives.

The base 2B model has not been fine-tuned on structured-products discourse and defaults to high-frequency continuations of “the investor ___” rather than to the domain-specific outcome implied by the prompt.

3.2 Per-position attribution shows the causal-chain tokens do contribute

Aggregate |attribution| from each input position to four selected logits (full table in top_features.md §2):

Two structural facts stand out.

First, the final token ( investor, position 18) dominates by a factor of ~3–5× for every output logit. This is structural to next-token attribution — the bulk of the computation that produces a prediction happens on the residual stream at the position whose next token is being predicted. It is not, on its own, evidence for or against domain reasoning.

Second, the barrier-related tokens are the next-strongest cluster. convertible (the product class), barrier at its second occurrence, and , (the syntactic boundary that delimits the conditional clause) all carry meaningful weight. below, closes, maturity, underlying, and the first barrier each contribute 4–9 units of |attribution| to the loses logit. The causal-chain tokens are not silent.

3.3 At the feature level the causal-chain signal is present but weak

top_features.md §4 lists features at concept-of-interest positions. For the loses logit, 98 such features clear the |attribution| > 0.05 threshold. The strongest:

0.140  L 2 pos11(' below')    feat=2916
0.137  L 0 pos 3(' barrier')  feat=12001
0.124  L 0 pos 3(' barrier')  feat=2612
0.122  L 8 pos13(' barrier')  feat=9505
0.116  L 1 pos13(' barrier')  feat=11094
0.112  L 3 pos10(' closes')   feat=5847
0.110  L 0 pos 3(' barrier')  feat=6224

A barrier-position feature path exists. It is order-of-magnitude weaker than the dominant features at the final-token position (~0.14 vs ~1.8 for the strongest single-feature contributor at L21 pos18) — which is consistent with the model representing the concept but not propagating it strongly enough to make loses the top prediction.

The same comparison for the rank-3 is logit (a generic verb-of-state continuation, not domain-aware) yields 38 concept-position features above threshold — about 2.5× fewer than loses. This is a small but interpretable directional signal: the loses logit’s path through the network involves the barrier-related input positions more than a content-free continuation does.

3.4 Honest disclosure: error-node dominance for loses

Of the top-15 source nodes for the loses logit, 8 are error nodes (CLT reconstruction residuals at the final-token position, layers 14–25). Error nodes are the part of the residual stream the transcoders failed to represent as discrete features. Their dominance means the actual computational mechanism for predicting loses is partially opaque to this attribution method. This is a known limitation of CLT-based interpretability and is exactly the kind of caveat an MRM reviewer must see in a per-decision documentation artefact.

4. FINMA 08/2024 Focus area VI mapping

FINMA’s findings letter for AI in financial services (08/2024) frames Focus area VI as follows (per docs/finma/08-2024-notes.md §2.6):

Observed: “results often cannot be understood, explained, or reproduced, and therefore cannot be critically assessed.”

Assessed: where decisions must be justified to investors, clients, employees, supervisor, or audit firm —

• Understanding the drivers of the applications
• Understanding behaviour under different conditions
• Assessing plausibility and robustness of results

This worked example is one concrete instance of the kind of artefact a firm could produce in response to “understanding the drivers of the applications” for a single LLM decision:

• Drivers identified. Section 3.2 lists every input token’s quantified contribution to the model’s choice of loses — a direct answer to “which input drove the output.”
• Per-feature decomposition. Section 3.3 names individual features at the relevant input positions and quantifies each one’s contribution. A reviewer can re-load the artefact (barrier_knockin_v1.pt) and inspect any node interactively.
• Reproducible. The script (mechinterp/scripts/03_attribution_graph.py), the prompt (mechinterp/prompts/barrier_knockin_v1.md), the model identifier, the transcoder set, the dtype, and the random-state-free attribution algorithm together fully determine the output. Given the same Spark / GB10 environment, anyone can regenerate barrier_knockin_v1.pt byte-for-byte deterministically.

What this artefact does not do, and an MRM reviewer should be told plainly:

1. One prompt is not one decision. A real SP advisory or pricing decision would have a longer prompt with structured retrieval-augmented context. The methodology generalises; the specific finding for this prompt does not.
2. CLT attribution is not ground-truth causality. Cross-layer transcoders factor the residual stream into a sparse basis; attribution along that basis is the model’s reconstructed computation, not a proven causal claim. The ~8 error nodes in §3.4 quantify exactly how much of the computation we are not explaining.
3. Gemma-2-2B is not a production model. No Swiss bank would deploy a 2B base model for SP work. This artefact demonstrates that the machinery exists at the small-scale end; replicating it on a frontier or production model is a separate, larger effort and may require different transcoder availability.

In FINMA’s vocabulary: this is the understanding the drivers leg of the §VI assessed-for list, demonstrated for one prompt. Behaviour under different conditions (the second leg) requires a population of prompts and is the Track B B1 work item. Plausibility and robustness of results (the third leg) requires comparison with expert expectations and a stability study across paraphrases — also Track B follow-on.

5. Caveats

• N = 1 prompt. No statistical claim is made; this is a feasibility demonstration.
• Cross-layer transcoders are an approximation to the residual-stream computation. Error-node dominance for the loses logit (§3.4) is a real limitation of this method on this prompt.
• Top-1 next token is receives, not loses — the base model’s surface behaviour is not domain-aware. The artefact’s value lies in the structural decomposition, not in the model’s correctness.
• GemmaScope feature interpretations are not auto-attached to feature IDs in this artefact. Each feat=NNNN triple (layer, pos, feat_id) would benefit from a Neuronpedia / GemmaScope auto-interp lookup; that is currently deferred (see mechinterp/neuronpedia/README.md).

6. Next step

The B1 design doc (docs/superpowers/plans/2026-05-b1-sae-drift-design.md, already drafted in Phase C) is the next plan: SAE-based drift monitoring across a population of SP prompts. That work answers FINMA §VI’s second assessment (“understanding behaviour under different conditions”) and converts this single-decision artefact into a population-level monitoring story.

A natural intermediate step before B1 execution is a contrastive variant of the present analysis: run the same attribution on Prompt 2 from mechinterp/prompts/barrier_knockin_v1.md (the negative case — underlying stays above the barrier, investor receives the coupon and full principal), and on Prompt 3 (neutral anchor). The difference between the positive and negative attributions would test whether the model’s barrier-concept features actually flip with the polarity of the conditional, which would be stronger evidence than the present single-prompt observation. Both prompts are already defined in the prompt file; only the attribution runs and a comparison writeup remain.