SAE and TensorLens: The Age of Feature Interpretability
Individual neurons are uninterpretable. Sparse Autoencoders extract monosemantic features from model internals, and TensorLens analyzes the entire Transformer as a single unified tensor.
SAE and TensorLens: The Age of Feature Interpretability
In the previous two posts, we:
- Logit/Tuned Lens: Read the model's intermediate predictions
- Activation Patching: Traced which activations are causally responsible for the answer
But here we hit a fundamental problem:
What do the activations we observe and manipulate actually *mean*?
Each dimension of an activation vector corresponds to an individual neuron. But these neurons are polysemantic -- a single neuron fires for academic citations, English dialogue, HTTP requests, and Korean text simultaneously. Clean interpretation at the neuron level is impossible.
This post covers two modern approaches that address this problem:
- Sparse Autoencoder (SAE): Decompose dense activations into sparse, monosemantic features
- TensorLens: Unify the entire Transformer computation into a single high-order tensor
Related Posts
Hybrid Mamba-Transformer, Measured: Qwen3.5-9B's Cache Is 4.4x Smaller Than Qwen3-8B's — and Why Our Speed Numbers Don't Count
Cache memory of Qwen3.5-9B (24 linear + 8 attention layers) vs Qwen3-8B measured from 2K to 64K context on one A100: 4.4x smaller at 64K, converging to 4.6x, and it decomposes exactly. Plus why HF eager speed numbers can't judge the architecture.
Paper of the Week #1 — Sparse Is Not the Same as Interpretable
I trained BDH and counted more than 4.8 billion activations across three controls. Training moved its single-latent sparsity from 49.98% to 81.65%, but a similar-budget ReLU baseline reached 91.04%. Sparsity is real; by itself, it is not evidence of interpretability.
Reproducing Claude's Watermark Locally — SynthID-Text on an Open Model, With Detection and Removal Attacks
We applied SynthID-Text — the algorithm Claude adopted — to Gemma 2 2B and measured everything: why detection is impossible without the key, how many tokens it needs, why false positives explode on short text, and how one rewrite by a local 3B model erases the watermark.