Learn AI by Building
From your first dataset to production agents — deep-dive series, hands-on notebooks, and experiments you can rerun yourself.
Paper of the Week
All Issues →Paper of the Week #7 — Told to Skip the Reasoning, It Kept Working in English and Mostly Stopped in Korean
This week's roundup. We gave Qwen3-8B the Thinking Inertia paper's "no reasoning, just the answer" instruction in English and in Korean: on 30 open-ended math questions it answered directly 0% of the time in English and 63.3% in Korean, where accuracy fell to 36.7%. Also: what a passing test guarantees, how much of an uncertainty score is answer length, agent cost tracking turns, who checks Lean's checkers, the conditions behind SSD-streaming speeds, decision models when their own server is at stake, three ways to write context down, and grading choices that reorder rankings.
Premium Series
Our Products
Courses and code built from what we measure here
Video Courses
13 hands-on courses — quantization, diffusion, RAG, on-device AI, decision models. $199 lifetime bundle, first 3 lectures of every course free
LLM Quantization and Compression Hands-On
GPTQ, AWQ, GGUF, QLoRA — fit LLMs into the memory you have. The course behind our KV-cache measurements
Free Benchmark Code
The scripts and raw results behind the measurement posts, starting with the KV cache harness. Free with an account
Premium Series
140 deep-dive posts across 21 series, bilingual KO/EN, with production-ready code and notebooks
Latest Posts
View All →
DeepSeek's Config Says 16 KV Heads, Its Cache Holds a Ninth of That: Seven Ways LLMs Handle Attention, and Which Models Use Each
DeepSeek-V2-Lite lists 16 KV heads, which predicts 8,640 MiB of cache at 32K; llama.cpp allocated 972 MiB. A guide to MHA, MQA, GQA, sliding window, MLA, sparse attention and linear-attention hybrids: what each stores, which models use it (checked against their config.json), and what it gives up.
Implementing the Attention Methods Used in LLMs: From Multi-Head Attention to Gated DeltaNet
MHA, MQA, GQA, sliding window, attention sinks, MLA with weight absorption, DSA-style sparse attention and Gated DeltaNet, each as a short PyTorch function with a check that it does what it claims. The MLA check runs on DeepSeek-V2-Lite's real layer-0 weights; the Qwen3.5-9B state size from the code matches llama.cpp's 50.25 MiB.
Four Times the Context, Less Memory: How Six Models Store the KV Cache, Measured in llama.cpp and vLLM
In llama.cpp, Qwen3.5-9B with a 128K context allocated 4,146 MiB of KV cache; Qwen3-8B needed 4,608 MiB for 32K. We calculated the cache from config.json for six models with four kinds of cache (full attention, sliding window, MLA, linear-attention hybrid), measured it in llama.cpp and vLLM, and list what to plan with for each.
llama.cpp vs vLLM: Deployment Methods and Strategies, Part 1: Memory
Deploying the same models on llama.cpp and vLLM, part 1: memory. What each runtime reserves for the KV cache and why (Gemma 3 4B at 32K: 814 MiB against 2,682 MiB), the settings that move it (-ub, --swa-full, max_num_batched_tokens, prefix caching), and four other differences: lengths past training, --parallel, GGUF conversion and vLLM's startup kernel build.
llama.cpp vs vLLM: Deployment Methods and Strategies, Part 2: Speed and Concurrent Requests
Measured on one A100 with Qwen3-8B in bf16 on both runtimes: one request generates at the same speed (86 tok/s), vLLM reads prompts faster, and at 64 concurrent requests vLLM produced 1,585 tok/s against llama.cpp's 430 (768 with its prompt cache off). Plus a llama.cpp default that stalled 64 requests for 144 seconds, what vLLM's lower memory reserve costs, and seven predictions scored.
Ten Minutes to Find Out Whether the Tests a Local Model Wrote for Your Function Protect a Bug
One script, two versions of your function, a local model: it asks the model for pytest tests from your current version only, runs every test against both versions, and tells you which tests would pass today and fail after the fix. On a date function with an off-by-one, Qwen3.5-9B wrote 10 tests in 6 seconds; 3 asserted the bug and none caught it. The same scorer and prompt as our 164-problem measurement.