TransformerLens in Practice: Reading Model Circuits with Activation Patching
Using TransformerLens to directly manipulate model activations, we trace which layers and heads causally produce the answer. A hands-on guide to activation patching.
TransformerLens in Practice: Reading Model Circuits with Activation Patching
In the previous post, we treated Lens as a window into the model's intermediate thoughts.
But "reading" alone cannot answer the most important question:
Does the model actually use this information?
Just because a hidden state at some layer contains "Paris" does not mean that layer causally contributes to the final answer. Information can be present but unused. A layer might hold the right answer in its representation, yet the model might arrive at its output through entirely different pathways.
To determine what actually matters, we need more than visualization. We need causal intervention: directly manipulating the model's internals and observing how the output changes.
1. TransformerLens: A Surgical Toolkit for Interpretability
TransformerLens is a mechanistic interpretability library created by Neel Nanda. Its core capability is attaching hooks to every internal activation in a Transformer, allowing you to read, modify, and replace activations at will.
pip install transformer_lensHookedTransformer: A Model Wired with Hooks
Related Posts
Paper of the Week #3 — Half the FLOPs Is Not Half the Time
One integer halves a fine-grained MoE's expert compute (arXiv 2609.04575) and its Table 5 replicates on one A100 to within a point. The paper never reports time, so I measured it: nothing in HF transformers, nothing at batch 1 in the vLLM you run today, 1.35x at batch 8. Plus the OLMoE control and the iso-cost harness control promised in issue #2.
The paper stopped at half the experts. A quarter loses four points instead of thirty, and a model trained without renormalization tells you why.
Two cells arXiv:2609.04575 never ran: k₁=2 and k₁=3 on Qwen3.6-35B-A3B, and the whole k₂ trick on OLMoE, which was trained without renormalization. The second one is a clean test of the paper's mechanism.
One integer halves MoE expert compute. We measured the speed the paper didn't: free at batch 8, not at batch 1.
We reproduced Table 5 of arXiv:2609.04575 on one A100 and measured the throughput the paper leaves out: nothing in HF transformers, nothing at batch 1 in stock vLLM, 1.35× at batch 8.