LLM Quantization and Compression
GPTQ, AWQ, bitsandbytes, GGUF, QLoRA, pruning and distillation — fit LLMs into the memory you have
What you'll learn
- ✓Explain why LLM inference is memory-bound and size a model's weights and KV cache
- ✓Understand activation outliers and how GPTQ and AWQ preserve accuracy at 4 bits
- ✓Use bitsandbytes (LLM.int8, NF4), convert models to GGUF and run them with llama.cpp
- ✓Fine-tune a 4-bit model with QLoRA and serve quantized models with vLLM or TGI
- ✓Apply unstructured and structured pruning and know when sparsity translates into speed
- ✓Combine distillation, pruning and quantization into one deployment pipeline
About this course
A 7B language model needs 28 GB in FP32. Naive INT8 breaks it because of activation outliers. And the tooling — GPTQ, AWQ, NF4, GGUF, QLoRA — is a zoo. This course sorts it out, with the math and with measurements.
We start with why LLMs are memory-bound and why weight-only quantization is the answer, then study activation outliers and the two algorithms that handle them: GPTQ (Hessian-guided error compensation) and AWQ (activation-aware scaling). Next the ecosystem: bitsandbytes LLM.int8 and NF4 with double quantization, GGUF and llama.cpp for CPU inference, QLoRA fine-tuning on a single GPU, and serving with vLLM and TGI. Then pruning — unstructured, structured, iterative with fine-tuning, and when it actually makes a model faster — and finally knowledge distillation and a complete compression pipeline as the final project: distill, prune, quantize, evaluate, export.
Every session is a notebook shown on screen with the narration explaining each cell's intent. Where a notebook's simulation is simplified relative to the real algorithm, or where a printed conclusion does not match the numbers, the narration says so — you will learn to read compression results critically.
Prerequisite: quantization basics (scale, zero-point, calibration, INT8), from our course *Neural Network Quantization in PyTorch and ONNX* or equivalent.
**Licensing.** Quantizing, pruning or distilling a model creates a derivative work that inherits the original licence — GPTQ, AWQ, GGUF and NF4 checkpoints included. The course ships a `Resources/LICENSES.md` with every asset checked against its actual licence text. Two to know now: **`facebook/opt-125m`, the small practice model, is research-only** (its model card declares `commercial: false`), and the Llama-2 and TheBloke checkpoints carry the Llama 2 Community License with its notice and 700M-MAU clauses. All the tooling — AutoGPTQ, AutoAWQ, bitsandbytes, llama.cpp, vLLM, TGI — is MIT or Apache-2.0 and constrains nothing.
Curriculum5 sections · 24 lectures · 3h 19m
Section 0. Welcome
Free previewSection 1. LLM Quantization GPTQ and AWQ
Free preview- ▶section 1 intro0:12
- ▶why llm quantization and activation outliers9:42
- 🔒gptq hessian error compensation9:24
- 🔒awq activation aware scaling8:42
- 🔒bit width group size and hub models10:30
- 🔒summary and exercises hessian gptq10:24
Section 2. GGUF bitsandbytes QLoRA and Serving
- 🔒section 2 intro0:12
- 🔒bitsandbytes llm int8 nf4 double quant12:30
- 🔒gguf and the llama cpp ecosystem9:12
- 🔒qlora fine tuning11:54
- 🔒serving with vllm tgi and choosing a method10:12
- 🔒exercises kv cache nf4 study and summary10:30
Section 3. Pruning
- 🔒section 3 intro0:12
- 🔒pruning intuition lottery ticket three dimensions9:00
- 🔒unstructured pruning random l1 weight orig mask10:48
- 🔒structured pruning channels neurons compaction10:24
- 🔒local vs global one shot vs iterative finetuning11:12
- 🔒random vs l1 curves summary exercises9:06
Section 4. Distillation Full Pipeline and Final Project
- 🔒section 4 intro0:06
- 🔒knowledge distillation10:42
- 🔒pruning quantization combination and nas11:06
- 🔒deployment pipeline and final project setup10:24
- 🔒final project walkthrough and course wrapup11:48
Requirements
- · A computer with Python 3.10+ (a free Google Colab account is enough for most sessions)
- · Comfort reading Python code; you do not need to be an expert
- · Quantization basics: scale, zero-point, calibration, INT8
- · Hugging Face transformers basics; a 16 GB+ GPU or Colab Pro for the LLM labs
- · Awareness that the practice model `facebook/opt-125m` is research-only and Llama-2 checkpoints are gated — see `Resources/LICENSES.md`
Who this is for
- · Engineers who need to run or fine-tune LLMs on limited GPUs or CPUs
- · ML practitioners choosing between GPTQ, AWQ, GGUF and bitsandbytes
- · Graduates of Neural Network Quantization in PyTorch and ONNX
- · Anyone who wants to read LLM compression papers and benchmarks critically
Read alongside the course
The voice-over in this course is synthesized with a text-to-speech model from scripts written and reviewed by the instructor, and the on-screen material (notebooks, code, slides) is the instructor's own work.