Diffusion LLM Part 4: LLaDA 2.0 -> 2.1 -- Breaking 100B with MoE + Token Editing
MoE scaling, Token Editing (T2T+M2T), S-Mode/Q-Mode, RL Framework -- how LLaDA 2.X makes diffusion LLMs practical.
Diffusion LLM Part 4: LLaDA 2.0 -> 2.1 -- Breaking 100B with MoE + Token Editing
In Part 3, LLaDA proved that "Diffusion LLMs are viable" by scaling Masked Diffusion to the 8B parameter range. But practical challenges remained: inference speed was far behind AR models, and alignment training like RLHF was absent.
In November 2025, Ant Group's InclusionAI began closing this gap with LLaDA 2.0. Then in February 2026, LLaDA 2.1 redefined the speed-quality tradeoff with an innovation called Token Editing.
This post covers the scaling journey from 8B to 100B, the adoption of MoE architecture, and how Token Editing works under the hood.
LLaDA 2.0: The Leap to 100B
LLaDA 2.0 shipped two models:
| Model | Total Params | Active Params | Layers | Heads | Context | Vocab |
|---|---|---|---|---|---|---|
| LLaDA 2.0-mini | 16B | 1.4B | 20 | 16 | 32,768 | 157,184 |
| LLaDA 2.0-flash | 100B | 6.1B | 32 | 32 | 32,768 | 157,184 |
The key change: introducing MoE (Mixture of Experts).
The original LLaDA 8B was a dense model -- every parameter activates for every input. LLaDA 2.0 adopts MoE, dramatically increasing total parameters while only activating a small subset of experts during inference.
LLaDA 2.0-flash activates just 6.1B of its 100B parameters. This is the same strategy used by AR MoE models like Mixtral and DeepSeek: "Keep the model's total knowledge broad, but keep inference costs low."
Related Posts
Paper of the Week #3 — Half the FLOPs Is Not Half the Time
One integer halves a fine-grained MoE's expert compute (arXiv 2609.04575) and its Table 5 replicates on one A100 to within a point. The paper never reports time, so I measured it: nothing in HF transformers, nothing at batch 1 in the vLLM you run today, 1.35x at batch 8. Plus the OLMoE control and the iso-cost harness control promised in issue #2.
The paper stopped at half the experts. A quarter loses four points instead of thirty, and a model trained without renormalization tells you why.
Two cells arXiv:2609.04575 never ran: k₁=2 and k₁=3 on Qwen3.6-35B-A3B, and the whole k₂ trick on OLMoE, which was trained without renormalization. The second one is a clean test of the paper's mechanism.
One integer halves MoE expert compute. We measured the speed the paper didn't: free at batch 8, not at batch 1.
We reproduced Table 5 of arXiv:2609.04575 on one A100 and measured the throughput the paper leaves out: nothing in HF transformers, nothing at batch 1 in stock vLLM, 1.35× at batch 8.