Diffusion LLM Part 3: LLaDA -- Building an 8B LLM with Masked Diffusion
Variable Masking, Fisher Consistency, In-Context Learning, Reversal Curse -- how LLaDA built a real LLM with diffusion.
Diffusion LLM Part 3: LLaDA -- Building an 8B LLM with Masked Diffusion
In Part 2, we explored how D3PM and MDLM define Diffusion in discrete spaces. We also confirmed that Absorbing State Diffusion using [MASK] tokens is the most effective approach for text.
However, prior work remained at relatively small scales. The question "Can we actually build a real LLM with Diffusion?" was answered by LLaDA (Large Language Diffusion with mAsking).
Nie et al. (2025) scaled Masked Diffusion to 8B parameters, directly compared it against LLaMA3 8B, and demonstrated that Diffusion LLMs can possess the core capabilities of AR models -- In-Context Learning and Instruction Following.
Core Idea: Variable Masking Ratio
The most important design decision in LLaDA is the variable masking ratio.
BERT masks a fixed 15% of the input during training. Once set, this ratio never changes.
LLaDA randomly samples the masking ratio from anywhere between 0% and 100% during training. In some batches, only 5% is masked; in others, 95% is masked.
Here is why this is critically important:
In-Context Learning: When the masking ratio is very low (e.g., 5%), the model predicts the remaining tokens while most tokens are already visible. This is essentially a "read the given context and fill in the blanks" task, which naturally connects to In-Context Learning.
Related Posts
Paper of the Week #3 — Half the FLOPs Is Not Half the Time
One integer halves a fine-grained MoE's expert compute (arXiv 2609.04575) and its Table 5 replicates on one A100 to within a point. The paper never reports time, so I measured it: nothing in HF transformers, nothing at batch 1 in the vLLM you run today, 1.35x at batch 8. Plus the OLMoE control and the iso-cost harness control promised in issue #2.
The paper stopped at half the experts. A quarter loses four points instead of thirty, and a model trained without renormalization tells you why.
Two cells arXiv:2609.04575 never ran: k₁=2 and k₁=3 on Qwen3.6-35B-A3B, and the whole k₂ trick on OLMoE, which was trained without renormalization. The second one is a clean test of the paper's mechanism.
One integer halves MoE expert compute. We measured the speed the paper didn't: free at batch 8, not at batch 1.
We reproduced Table 5 of arXiv:2609.04575 on one A100 and measured the throughput the paper leaves out: nothing in HF transformers, nothing at batch 1 in stock vLLM, 1.35× at batch 8.