Neural Network Quantization in PyTorch and ONNX
Number formats, quantization math, dynamic/static/QAT, ResNet and YOLO on ONNX Runtime — measured, not assumed
What you'll learn
- ✓Explain FP32/FP16/BF16/INT8 representations and predict overflow and precision-loss behaviour
- ✓Derive scale and zero-point for affine and symmetric quantization and choose a calibration method
- ✓Apply dynamic and static quantization in PyTorch, including observers, QuantStub and module fusion
- ✓Run quantization-aware training and know when it beats post-training quantization
- ✓Quantize a pretrained ResNet, handle skip connections with FloatFunctional, and measure size and latency
- ✓Export YOLOv8 to ONNX and quantize it with ONNX Runtime; understand ConvInteger vs QLinearConv pitfalls
About this course
Quantization is the most effective way to make a trained model smaller and faster — and the easiest way to silently destroy its accuracy. This course teaches it from first principles and measures everything.
We start with how computers represent numbers (FP32, FP16, BF16, INT8) and the actual math of quantization: scale, zero-point, symmetric vs asymmetric, per-tensor vs per-channel, calibration methods (min-max, percentile, MSE, KL) and error analysis. Then we implement the three PyTorch workflows — dynamic quantization, static quantization with observers and module fusion, and quantization-aware training with FakeQuantize and the straight-through estimator — on a CNN, and compare accuracy, size and latency honestly. Finally we quantize a pretrained ResNet-18 on CIFAR-10 (including the skip-connection problem) and export YOLOv8 to ONNX to quantize with ONNX Runtime for CPU and edge deployment.
Every session is a Jupyter notebook shown on screen; the narration explains the intent of each cell and calls out where results are surprising or where a notebook's own printed conclusion does not match the numbers — because that is how real quantization work looks. Each session ends with a decision rule: which technique, for which model, on which hardware.
The sequel course, *LLM Quantization and Compression*, covers GPTQ, AWQ, GGUF, QLoRA, pruning and distillation.
**Licensing — one thing to know before session 7.** **Ultralytics YOLOv8 is AGPL-3.0.** If you take the YOLO workflow into a product and distribute it, or expose it over a network, AGPL-3.0 requires you to publish the complete source of your entire application — Ultralytics states this extends to model weights — unless you buy their Enterprise licence. Sessions 1 to 6 train models you own on MNIST and CIFAR-10 and constrain nothing. The course ships a `Resources/LICENSES.md` with every asset checked against its actual licence text, and names permissively licensed detectors you can substitute without changing a single quantization step.
Curriculum8 sections · 41 lectures · 5h 39m
Section 0. Welcome
Free previewSection 1. Number Representation
Free preview- ▶section 1 intro0:12
- ▶why model compression11:30
- 🔒ieee754 fp32 fp16 bf1612:24
- 🔒integers int8 overflow8:48
- 🔒what is quantization15:18
Section 2. Quantization Math
- 🔒section 2 intro0:12
- 🔒affine quantization scale zero point9:48
- 🔒symmetric per tensor per channel10:30
- 🔒calibration methods9:24
- 🔒quantization error analysis10:00
Section 3. Dynamic Quantization
- 🔒section 3 intro0:12
- 🔒ptq and how dynamic quantization works10:30
- 🔒inside dynamicquantizedlinear and mnist cnn setup10:36
- 🔒quantizing the cnn results structure and limits10:48
- 🔒dynamic quantization for transformers10:36
- 🔒summary exercises and road to static10:00
Section 4. Static Quantization
- 🔒section 4 intro0:12
- 🔒static quantization principles10:54
- 🔒module fusion and training8:42
- 🔒calibration and convert9:30
- 🔒evaluation and observer comparison9:48
- 🔒common errors and skip connections9:06
Section 5. Quantization-Aware Training
- 🔒section 5 intro0:06
- 🔒why qat and how fakequantize works10:00
- 🔒fakequantize and ste in code8:54
- 🔒full qat workflow on mnist11:18
- 🔒dynamic vs static vs qat and practical tips10:12
- 🔒exercises lr sweep convergence custom fakequantize and summary10:24
Section 6. Quantizing ResNet
- 🔒section 6 intro0:12
- 🔒pretrained strategy and skip connections8:36
- 🔒fp32 baseline and dynamic quantization10:18
- 🔒static quantization error and quantizable resnet10:06
- 🔒qat finetune and four way comparison10:12
- 🔒saving loading and deployment9:54
Section 7. YOLO and ONNX Runtime
- 🔒section 7 intro0:12
- 🔒pytorch quantization limits and why onnx9:12
- 🔒onnx export and onnxruntime inference8:30
- 🔒onnxruntime dynamic and static quantization10:24
- 🔒yolov8 onnx int8 end to end9:24
- 🔒detection results deployment runtimes summary exercises10:24
Requirements
- · A computer with Python 3.10+ (a free Google Colab account is enough for most sessions)
- · Comfort reading Python code; you do not need to be an expert
- · Basic PyTorch (models, training loops)
- · A GPU helps for sessions 6–7 but is not required
- · Awareness that session 7 uses Ultralytics YOLOv8, which is AGPL-3.0 — see `Resources/LICENSES.md` before using it commercially
Who this is for
- · ML engineers deploying models to CPUs, mobile or edge devices
- · Data scientists who need smaller, faster models without accuracy surprises
- · Students who want the math behind INT8 rather than a one-line API call
- · Anyone preparing for the LLM quantization course
Read alongside the course
The voice-over in this course is synthesized with a text-to-speech model from scripts written and reviewed by the instructor, and the on-screen material (notebooks, code, slides) is the instructor's own work.