Karpathy's microgpt.py Dissected: Understanding GPT's Essence in 150 Lines
A line-by-line dissection of microgpt.py -- a pure Python GPT implementation with zero dependencies. Training, inference, and autograd in 150 lines.
Karpathy's microgpt.py Dissected: Understanding GPT's Essence in 150 Lines
Andrej Karpathy has released new code. This time, it is even more extreme than nanoGPT. A 150-line script that trains and runs inference on a GPT, using pure Python with no external libraries.
No PyTorch. No NumPy. Just three imports: os, math, random.
The comment at the top of the code says it all:
"This file is the complete algorithm. Everything else is just efficiency."
In this post, we dissect microgpt.py line by line. Follow along with the code, and you will see that the algorithm behind GPT is a surprisingly simple composition of mathematical operations.
Overall Structure
microgpt.py breaks down into roughly 6 parts:
| Part | Lines | Role |
|---|---|---|
| Data & Tokenizer | ~10 | Load name dataset, character-level tokenization |
| Value Class (Autograd) | ~35 | Scalar automatic differentiation engine |
| Parameter Initialization | ~15 | Weight matrix creation (4,192 parameters) |
| Model Architecture | ~40 | Embedding + Attention + MLP + RMSNorm |
| Training Loop | ~20 | Cross-entropy loss + Adam optimizer |
| Inference | ~15 | Name generation via temperature sampling |
Total parameters: 4,192. Compared to GPT-2 Small's 124M, that is roughly 30,000x smaller. But the algorithm is identical.
Related Posts
Paper of the Week #3 β Half the FLOPs Is Not Half the Time
One integer halves a fine-grained MoE's expert compute (arXiv 2609.04575) and its Table 5 replicates on one A100 to within a point. The paper never reports time, so I measured it: nothing in HF transformers, nothing at batch 1 in the vLLM you run today, 1.35x at batch 8. Plus the OLMoE control and the iso-cost harness control promised in issue #2.
The paper stopped at half the experts. A quarter loses four points instead of thirty, and a model trained without renormalization tells you why.
Two cells arXiv:2609.04575 never ran: kβ=2 and kβ=3 on Qwen3.6-35B-A3B, and the whole kβ trick on OLMoE, which was trained without renormalization. The second one is a clean test of the paper's mechanism.
One integer halves MoE expert compute. We measured the speed the paper didn't: free at batch 8, not at batch 1.
We reproduced Table 5 of arXiv:2609.04575 on one A100 and measured the throughput the paper leaves out: nothing in HF transformers, nothing at batch 1 in stock vLLM, 1.35Γ at batch 8.