AI Research•February 6, 2026•KR

From Logit Lens to Tuned Lens: Reading the Intermediate Thoughts of Transformers

What happens inside an LLM between input and output? Logit Lens and Tuned Lens let us observe how Transformers build predictions layer by layer.

From Logit Lens to Tuned Lens: Reading the Intermediate Thoughts of Transformers

From Logit Lens to Tuned Lens: Reading the Intermediate Thoughts of Transformers

You type "The capital of France is" into an LLM and get back "Paris." But *where* inside the model did that answer actually form?

TL;DR

Logit Lens projects intermediate hidden states to vocabulary space using the model's final unembedding matrix
This reveals how Transformers build predictions incrementally, layer by layer
Tuned Lens fixes Logit Lens's systematic bias by learning a lightweight affine transformation per layer
Together, these tools give us a principled way to peek inside the black box

1. Where Does the Answer Come From?

When you ask GPT "What is the capital of France?", the answer "Paris" comes back in milliseconds. But think about it: the model has 32, 48, or even 96 layers. Did it "know" the answer at Layer 1? Did it only figure it out at the very last layer? Or did it gradually build up confidence somewhere in the middle?

This is not just a philosophical question. If we could watch the model's internal state evolve from layer to layer, we would gain real insight into *how* these models think. We could debug failures, detect hallucinations, and understand what each layer actually contributes.

That is exactly what Logit Lens and Tuned Lens let us do.

2. Residual Stream: The Information Highway

Before we can understand how to "read" intermediate layers, we need to understand how information flows through a Transformer.

A Transformer does not pass information from one layer to the next like a conveyor belt. Instead, every layer reads from and writes to a shared vector called the residual stream. Think of it as a shared whiteboard that every layer can scribble on.

Mathematically, the hidden state at layer $l$ is:

🔒

Sign in to continue reading

Create a free account to access the full content.

Sign In / Sign Up

Related Posts

InternVL-U: Understanding + Generation + Editing in One 4B Model -- A New Standard for Unified Multimodal AI

InternVL-U: Understanding + Generation + Editing in One 4B Model -- A New Standard for Unified Multimodal AI

Shanghai AI Lab's InternVL-U. A single 4B parameter model handles image understanding, generation, editing, and reasoning-based generation. Decoupled visual representations outperform 14B BAGEL on GenEval and DPG-Bench.

Hybrid Mamba-Transformer MoE: Three Teams, One Architecture -- The 2026 LLM Convergence

Hybrid Mamba-Transformer MoE: Three Teams, One Architecture -- The 2026 LLM Convergence

NVIDIA Nemotron 3 Nano, Qwen 3.5, and Mamba-3 independently converge on 75% linear layers + 25% attention + MoE. 88% KV-cache reduction, O(n) complexity for long-context processing.

Spectrum: 3-5x Diffusion Speedup Without Any Training -- The Power of Chebyshev Polynomials

Spectrum: 3-5x Diffusion Speedup Without Any Training -- The Power of Chebyshev Polynomials

CVPR 2026 paper from Stanford/ByteDance. Chebyshev polynomial feature forecasting achieves 4.79x speedup on FLUX.1, 4.56x on HunyuanVideo. Training-free, instantly applicable to any model.