Latent Diffusion to SANA

What you'll learn

• ✓Explain why latent diffusion is faster and how the VAE, scaling factor and two-stage training fit together
• ✓Implement text-conditional latent diffusion with CLIP embeddings and cross-attention
• ✓Implement classifier-free guidance for training (condition dropout) and sampling (guidance scale)
• ✓Build a Diffusion Transformer with adaLN-Zero conditioning and train it with class and CLIP conditions
• ✓Understand PixArt-α's multimodal DiT block and reproduce it on MNIST latents
• ✓Implement SANA's linear attention block and rectified-flow (flow matching) training and few-step sampling

About this course

This course picks up where DDPM ends and follows the field to the models that dominate today. We move diffusion into a compressed latent space with a VAE (the idea behind Stable Diffusion), add text conditioning with CLIP and classifier-free guidance, replace the U-Net with a Transformer (DiT), and finish with the efficient architectures of 2024: PixArt-α and SANA with linear attention and rectified flow.

Each chapter follows the same recipe: read the paper, derive the key equations in plain language, implement the model in PyTorch, train it on MNIST latents, and look at the samples. You will implement a two-stage latent pipeline, a text-conditional U-Net with cross-attention, CFG training with condition dropout and CFG sampling, DiT blocks with adaLN-Zero, CLIP-conditioned DiT, the multimodal DiT of PixArt, and SANA's linear attention with a rectified-flow scheduler.

Every lecture shows the notebook on screen while the narration explains each cell. Trade-offs and failure modes are discussed openly, and every model is compared with the one before it so you understand what each paper actually changed.

Prerequisite: DDPM and DDIM basics — from our course *DDPM and DDIM from Scratch* or equivalent.

**Licensing.** The models you train here are yours, but two pretrained components are downloaded and frozen: `stabilityai/sd-vae-ft-mse` (MIT — no obligations beyond attribution) and `openai/clip-vit-base-patch32`, whose weights carry no declared licence and whose model card states that any deployed use, commercial or not, is out of scope. `Resources/LICENSES.md` sets this out and points at the fully permissive T5 conditioning path that sessions 5 and 6 already use.

Curriculum5 sections · 23 lectures · 2h 37m

Read alongside the course

• From 512×512 to 1024×1024: How Latent Diffusion Broke the Resolution Barrier
• DiT: Replacing U-Net with Transformer Finally Made Scaling Laws Work (Sora Foundation)
• SANA: O(n²)→O(n) Linear Attention Generates 1024² Images in 0.6 Seconds