← all analysis · · docs/analysis/2026-05-nla-sidecar.md

NLA Sidecar — Cross-method Comparison on Barrier Knock-in

On this page

Date: 2026-05-17 Status: Complete — Path A Week 1 deliverable. SGLang served the released NLA actor on the GB10 Spark (Triton attention backend; flashinfer crashes during warmup on Gemma-3); Prompts 1/2/3 were captured against live Gemma-3-12B-IT L32 activations and decoded through kitft/nla-gemma3-12b-L32-av. §3/§4 below are filled from that local run — the Neuronpedia fallback (§2) was not needed. Scope note: §2 documents the Neuronpedia walkthrough kept as the fallback recipe. The path actually taken was local SGLang on the Spark — see §3 for the pipeline and §7 for the run record. Companion artefact: /analysis/2026-05-first-attribution-graph/ (the Gemma-2-2B + GemmaScope attribution graph this doc compares against).

This sidecar tests one question: given the same SP-domain prompt, does an independent interpretability method produce a narrative consistent with what the attribution-graph method said? Method-level triangulation is the FINMA §VI third-leg requirement (plausibility / robustness of results); a single-method explanation cannot satisfy it.

1. Setup

Reference NLA paper: Anthropic, Natural Language Autoencoders for LLM Interpretability, transformer-circuits.pub, 2026 — https://transformer-circuits.pub/2026/nla/index.html. Code: https://github.com/kitft/natural_language_autoencoders (Apache-2.0). Released checkpoints (HuggingFace kitft/nla-models): Qwen-2.5-7B L20, Gemma-3-12B L32, Gemma-3-27B L41, Llama-3.3-70B L53. Browsable UI: https://www.neuronpedia.org/nla.

Why Gemma-3-12B. Closest to the Phase B model family (still Gemma), largest available checkpoint that fits the Spark’s memory budget if step 2 (local serving) is reached, and not so large that the cross-scale gap to 2B becomes the only story.

Prompt. Prompt 1 from mechinterp/prompts/barrier_knockin_v1.md:

“In a barrier reverse convertible, if the underlying closes below the barrier at maturity, the investor”

Same prompt used in /analysis/2026-05-first-attribution-graph/. Single token to predict; the residual-stream activation at L32 over the final-token position is what the NLA verbalizes.

What we are NOT doing.

2. Method (step 1 — Neuronpedia)

  1. Navigate to neuronpedia.org/nla, select the Gemma-3-12B L32 NLA.
  2. Submit Prompt 1 verbatim. Capture the verbalized AV explanation for the residual-stream activation at L32, position of the final token ( investor).
  3. Record any auxiliary outputs the UI surfaces — top contributing tokens, related activations, intervention previews. Note which are inference-time outputs of the released NLA vs. pre-computed metadata.
  4. Re-run with paraphrases of Prompt 1 (one or two minor rewordings) to get a stability read on the verbalization.

(Optional, only if step 1 produces something concrete enough to extend.) Repeat for Prompt 2 (above-barrier) and Prompt 3 (neutral anchor). This step matters: a verbalization that does not flip with the polarity of the conditional is evidence the NLA is template-matching the surface, not the causal claim.

3. Captured outputs

Source of record: local SGLang AV decode on the GB10 Spark — not Neuronpedia. Pipeline: mechinterp/scripts/04_capture_l32_activations.py captures the L32 residual-stream activation (Gemma-3-12B-IT, output_hidden_states index 32, final-token position) → mechinterp/scripts/nla_inference.py injects it into the released kitft/nla-gemma3-12b-L32-av actor via SGLang input_embeds. Raw artefacts under mechinterp/outputs/nla_sidecar_v1/ (l32_activations.parquet, l32_capture_meta.json, av_decode_v1.txt). AV decode at temperature 0; the run was repeated and was byte-identical (deterministic).

Activation L2-norms ‖act‖ are 66k–80k — inside the NLA’s training band (the released Gemma-3-12B checkpoint expects residual norms near its injection_scale of 80 000; the actor rescales every input to exactly that norm before injection, so the raw norms only need to be the right order of magnitude — they are).

3.1 Prompt 1 (positive — below barrier)

“In a barrier reverse convertible, if the underlying closes below the barrier at maturity, the investor”

3.2 Prompt 2 (negative — above barrier)

“In a barrier reverse convertible, if the underlying stays above the barrier at maturity, the investor”

3.3 Prompt 3 (neutral anchor)

“A barrier reverse convertible is a structured product”

3.4 Logit-level directionality (cross-check)

The next-token logits carry a weak but correctly-signed directional signal that the top-1 token hides: loses is ~13× more probable below-barrier (p=0.069) than above-barrier (p=0.005). Phase B reported the analogous Gemma-2-2B fact (top-1 receives, loses far down the list). The directionality exists at the logit margin in both model generations; the L32 NLA verbalization surfaces it explicitly for the below-barrier case (§3.1).

4. Cross-method comparison

The Phase B artefact (2026-05-first-attribution-graph.md §3) reported:

The NLA verbalization (§3 above) answers a different question: what English-language description of the L32 activation reconstructs that same activation under the AR? Comparison axes:

AxisPhase B attribution graphNLA sidecar
ModelGemma-2-2B (base, 2B, 26 layers)Gemma-3-12B-IT (instruction-tuned, 12B, 48 layers)
SitePer-feature, per-position, all layersSingle residual-stream site at L32, final-token position
Output unitSparse feature IDs + scalar attributionsNatural-language paragraph
DecompositionDiscrete (CLT features + error nodes)Holistic (a verbalization of the whole activation)
Causal validationGradient-based attribution to logitsRL-trained reconstruction objective + (paper) steering
Cross-modelLocked to Gemma-2 + GemmaScope availabilityLocked to released NLA model list

Convergence test: does the NLA verbalization mention the same input-token cluster the attribution graph identified — barrier, below, underlying, maturity, the conditional structure? A “yes” is weak corroboration (both methods see the prompt’s structurally salient content, which neither would miss if the model were just template-matching). A “no” — verbalization speaks only about the surface phrase “the investor” and produces no mention of the conditional — is interesting: it would suggest the L32 activation in the larger Gemma-3-12B carries less of the causal-chain content than Phase B’s per-position attributions on Gemma-2-2B do. Either outcome is reportable.

Divergence test: does the NLA produce a domain-aware claim about “loses principal protection” or “receives shares below par,” when Gemma-2-2B’s surface output preferred the generic receives continuation? If yes, the larger instruction-tuned model has the domain content available even when its generation defaults elsewhere — relevant prior for any production-class model audit.

Findings (2026-05-17)

Convergence — yes. The NLA verbalizations name the same concept cluster the Phase B attribution graph flagged: barrier, structured product, payoff, principal, and — for Prompt 1 — the below-barrier → loss conditional. Both methods see the prompt’s structurally salient content. As the convergence test above notes, this is weak corroboration: neither method would miss this content if the model were merely template-matching the surface phrase.

Divergence — partial, and informative. The divergence test asked whether the larger instruction-tuned model produces a domain-aware loss-of-protection claim where Gemma-2-2B’s surface output defaulted to the generic receives. The answer is a qualified yes for the below-barrier prompt: the L32 verbalization explicitly says “the loss of principal or return” even though Gemma-3-12B-IT’s own top-1 continuation is receives (p=0.845) — the domain content is in the L32 activation, just not in the dominant logit. For the above-barrier prompt the verbalization is equivocal — it does not cleanly flip to the principal-protected reading. So the L32 activation encodes the below → loss leg more crisply than the above → protected leg. This asymmetry is the single most citable finding of the sidecar, and it sharpens — without contradicting — the Phase B contrastive result (2026-05-16-attribution-graph-contrastive.md), which found Gemma-2-2B base had no directional barrier representation at all.

Caveat — confabulation, exactly as the source paper warns. Every verbalization mis-names the product: “equity linked note”, “Enhanced Yield Note”, “ETF/bond hybrid” — the prompt says barrier reverse convertible. These are thematically faithful (all structured products) but verifiably false on the specific instrument. Per §5 and Anthropic’s NLA caveats, the direction claim (“below the barrier → loss of principal”) is treated as a hypothesis — corroborated here by the prompt text it quotes and by the independent logit-margin cross-check in §3.4, not asserted as a standalone finding. The product mis-naming is logged as a concrete confabulation instance for the Week 6 limitations section.

5. Caveats

6. FINMA 08/2024 §VI mapping

docs/finma/08-2024-notes.md §2.6 lists the three assessment legs: drivers, behaviour-under-conditions, plausibility/robustness. Phase B answered the first leg with one method on one prompt. This sidecar contributes to the third leg: plausibility and robustness of results, by showing whether an independent verbalization method tells a story consistent with the attribution-graph story.

What this artefact does not do:

  1. It does not produce regulator-grade evidence on its own. Two methods agreeing on three prompts is a feasibility demonstration of the triangulation pattern, not a sign-off-grade audit.
  2. It does not replace the population-level work in B1 (docs/superpowers/plans/2026-05-b1-sae-drift-design.md). That remains the answer to behaviour under different conditions.
  3. It does not address whether NLAs would be deployable as a runtime monitor — the inference cost (hundreds of tokens per activation) makes that a different question with a different cost model.

7. Step 2 (local SGLang) — run record

Done 2026-05-17. Step 1 (Neuronpedia) was skipped — the local path was taken directly and succeeded. Run record:

  1. SGLang on GB10 — passed. sglang 0.5.12 installs from an aarch64 wheel. The default flashinfer attention backend installs and starts the server but crashes during model warmup (q.shape[0] does not match qo_indptr[-1] — a flashinfer paged-prefill shape bug on Gemma-3). --attention-backend triton serves end to end. Launch also needs the venv and CUDA toolchain on PATH (flashinfer JIT-compiles via ninja/nvcc) and --disable-radix-cache (mandatory for input_embeds requests).
  2. Memory. Sequential pipeline per the Path A plan: stop inference-server-vllm-1, capture activations with the target model, free it, then serve the AV. Both the 24 GB target and the 22 GB AV fit the 121 GB Spark with margin when run sequentially.
  3. Client patch. nla_inference.py (vendored at mechinterp/scripts/) needed one fix: apply_chat_template(tokenize=True) returns a BatchEncoding under transformers ≥5.0, not a token-id list — return_dict=False restores the list the client expects.
  4. Outputs. §3.1–§3.3 above are the locally-served verbalizations. No Neuronpedia comparison was run, so no hosted-vs-local divergence check — a possible follow-up if the artefact is extended.

Original conditional framing (kept for context): if step 1 produced a verbalization concrete enough to compare meaningfully, the next step was to reproduce it under our control:

  1. SGLang compatibility check on GB10. SGLang’s ARM/Grace support is uneven. Run a minimal SGLang python -m sglang.launch_server --model-path google/gemma-3-12b-it on the Spark and verify it serves before downloading NLA weights. If it fails, the sidecar stops at step 1 and we cite Neuronpedia as the source of record.
  2. Memory budget. Same Spark constraint as the rest of Track B: stop inference-server-vllm-1 first; check free -h for ≥30 GB free; release with docker stop and restart after.
  3. Run order. AV inference per prompt → record output. Optionally AR round-trip to confirm the released checkpoints reconstruct the original activation within published FVE bounds — but only as a sanity check, not a finding.
  4. Write-up extension. Append §3.1–§3.3 with the locally reproduced verbalizations and any divergence from the Neuronpedia outputs. Divergence between Neuronpedia-hosted and locally-served versions would be a process finding worth flagging.

8. Status