IntermediateEnglish19 lectures · 2h 33m

On-Device LLMs Model Preparation

Mobile hardware, compression theory, choosing a small model, and converting it with llama.cpp, MLC LLM and ONNX Runtime

Watch 3 lectures free

What you'll learn

  • ✓Explain why on-device inference is memory-bandwidth-bound and estimate tokens per second from hardware specs
  • ✓Apply the quantization formula and choose between PTQ, QAT, pruning and distillation
  • ✓Select a small language model with a repeatable framework, including KV-cache sizing and tokenizer efficiency
  • ✓Build llama.cpp, convert to GGUF and compare bit widths with real benchmarks
  • ✓Compile a model with MLC LLM and convert one to ONNX Runtime with INT4 weights

About this course

Before a language model can run on a phone it has to be chosen, shrunk and converted — and every one of those decisions has a measurable consequence. This course is the model half of on-device AI.

We start with the hardware: what is inside a mobile SoC, why memory bandwidth rather than compute is the wall, and how the Roofline model tells you which optimisations will matter. Then compression theory: the quantization formula, post-training quantization versus quantization-aware training, pruning, distillation, and the modern LLM methods (GPTQ, AWQ, SmoothQuant, bitsandbytes) with a decision tree for choosing among them. Then the practical path: selecting a small language model with a repeatable framework (memory budget, KV cache, tokenizer efficiency for your target language, benchmarks), building llama.cpp and quantizing to GGUF at four bit widths with benchmarks, compiling with MLC LLM for a mobile GPU, and converting with Hugging Face Optimum to ONNX Runtime with INT4 weight-only quantization.

Every session is a notebook shown on screen while the narration explains what each command does and what the output means. The follow-up course, *Ship an LLM in an Android App*, takes the model you build here into Kotlin and React Native apps.

**Licensing.** Converting a model to GGUF, MLC or ONNX creates a derivative work that inherits the original licence, so this course ships a `Resources/LICENSES.md` with every model checked against its actual licence text — including one correction to the notebooks: **`Qwen2.5-3B-Instruct` is under the Qwen Research License, "FOR NON-COMMERCIAL PURPOSES ONLY", not Apache-2.0** as session 3's comparison table states. Gemma 2 and Llama 3.2 are commercially usable with notice and attribution duties; Phi-3-mini (MIT) and TinyLlama (Apache-2.0) are the unencumbered options.

Curriculum7 sections · 19 lectures · 2h 33m

Section 0. Welcome

Free preview

Section 1. On-Device AI and Mobile Hardware

Free preview

Section 2. Model Compression Theory

  • 🔒quantization formula and ptq10:00
  • 🔒qat ste pruning distillation9:36
  • 🔒llm quantization and decision tree9:12

Section 3. Small Language Models

  • 🔒section 3 intro0:12
  • 🔒slm definition and lineup9:30
  • 🔒kv cache and tokenizer efficiency9:12
  • 🔒language benchmarks and selection framework9:36

Section 4. llama.cpp and GGUF

  • 🔒environment token and cmake build9:30
  • 🔒download convert quantize and bench8:30
  • 🔒size vs speed quality and tuning8:30

Section 5. MLC LLM

  • 🔒mlc difference install convert11:54
  • 🔒mlc compile artifacts test port8:36

Section 6. ONNX Runtime Conversion

  • 🔒why onnx optimum export8:54
  • 🔒int4 woq inference execution providers9:12
  • 🔒ep benchmark graph speculative app tips9:24

Requirements

  • · A computer with Python 3.10+ (a free Google Colab account is enough for most sessions)
  • · Comfort reading Python code; you do not need to be an expert
  • · A Hugging Face account for model downloads
  • · A GPU helps; most notebooks run on CPU or Colab
  • · Willingness to read `Resources/LICENSES.md`: several of the models compared here are gated, and Qwen2.5-3B is non-commercial

Who this is for

  • · Mobile and ML engineers preparing models for on-device deployment
  • · Developers choosing between GGUF, MLC and ONNX Runtime
  • · Anyone who wants measured answers about on-device model performance

Read alongside the course

The voice-over in this course is synthesized with a text-to-speech model from scripts written and reviewed by the instructor, and the on-screen material (notebooks, code, slides) is the instructor's own work.

On-Device LLMs Model Preparation | SOTAAZ Blog