Learn AI by Building
From your first dataset to production agents — deep-dive series, hands-on notebooks, and experiments you can rerun yourself.
Tutorials
View All →LLM Agent Cookbook
Build AI agents from scratch — ReAct, Tool Use, Multi-Agent orchestration
ML Cookbook
Master machine learning algorithms with hands-on Jupyter projects
Data Analysis Cookbook
SQL, Pandas, Statistics — everything for data-driven decisions
Ontology & KG Cookbook
RDF, OWL, Neo4j, and GraphRAG for knowledge-powered AI
Paper of the Week
All Issues →Paper of the Week #3 — Half the FLOPs Is Not Half the Time
One integer halves a fine-grained MoE's expert compute (arXiv 2609.04575) and its Table 5 replicates on one A100 to within a point. The paper never reports time, so I measured it: nothing in HF transformers, nothing at batch 1 in the vLLM you run today, 1.35x at batch 8. Plus the OLMoE control and the iso-cost harness control promised in issue #2.
Premium Series
Our Products
Courses and starter kits built from what we measure here
Video Courses
13 hands-on courses — quantization, diffusion, RAG, on-device AI, decision models. $199 lifetime bundle, first 3 lectures of every course free
LLM Quantization and Compression Hands-On
GPTQ, AWQ, GGUF, QLoRA — fit LLMs into the memory you have. The course behind our KV-cache measurements
Starter Kits
Solution notebooks for the free cookbooks — LLM Agent, Data Analyst, ML, Ontology & KG. One-time purchase
Premium Series
140 deep-dive posts across 21 series, bilingual KO/EN, with production-ready code and notebooks
Starter Kits
View All →Practice notebooks, interview questions, and project solutions — ready to download.
Browse Starter KitsLatest Posts
View All →
How Many Labels Is a Decision Model Worth? Kev on Three Tasks It Never Saw
On CLINC150, MASSIVE and financial-news tweets, none of which Kev trained on, Kev-9B matched a small classifier trained on roughly 2 to 20 labelled examples per class. Its probabilities ran too low: stated confidence sat 10–17 points under its accuracy (calibration error 11–17%, against 2–9% on familiar data), so a 95%-accuracy threshold passed only 23–58% of messages. Out-of-scope questions got low probabilities: 5 of 100 passed that threshold. The 0.8B model answered 'Financials' to 152 of 200 tweets.
Kev vs MoJev: Two More Open Jev Alternatives, Accuracy and Confidence Measured
Kev (0.8B to 9B) and MoJev (0.85B) both speak Jev's API. On BANKING77, TREC and AG News with human labels, Kev's probabilities were usable: at 95% accuracy it could auto-accept 53–61% of BANKING77, where laya managed 0%. But Kev was trained on these three datasets, and a small classifier trained on the same data still beat it on BANKING77. MoJev reports 0.79% calibration error; here it was 6–22%, with probabilities too low.
Can You Trust a Decision Model's Confidence? Repeated Answers and Probabilities, Measured
Decision models sell a probability with every answer, so you can act on the sure ones and escalate the rest. I checked both ways of judging 'sure' on human-labelled data. Across 8 noise draws, 77–80% of openjev's wrong answers were unanimous. Probabilities did better, but how useful they were varied sharply by task: laya could auto-accept 88% of TREC at 95% accuracy and 0% of BANKING77. The new CLM-8B stayed near chance on all three datasets when given label names.
Jev in 10 Minutes: What a Fast Decision Model Is, and When It Is Worth Using
Jev answers a question with a probability for each option instead of writing text. What that is for, how open projects do the same thing, and what three rounds of measurement showed: with a handful of options a 421M model reached 86.6% in 23 ms, with 77 a small trained classifier on a CPU still led at 90%.
Open-Source Jev Alternatives, Measured: laya, openjev and NanoJev on Three Public Datasets
I ran laya, openjev and NanoJev on BANKING77 (77 intents), TREC (6 question types) and AG News on one A100. With six options, laya's 421M encoder reached 86.6% on TREC in 23 ms. With 77, the DiffusionGemma-based openjev led at 66.9%, and a logistic regression on a CPU beat everything at 90%.
DiffusionGemma on vLLM's Example Server: One Sentence of Context Took It From 54.5% to 31.8%
I ran the example server from the merged vLLM pull request that turns DiffusionGemma into a decision model on 154 BANKING77 messages. It caps a question at 26 options, so I split the 77 intents into nine groups and read twice: 54.5% at 119 ms with one noise draw, 51.3% with the default draws. My transformers version of the same two-stage read scored 68.2%, and one added sentence describing the input dropped the server from 54.5% to 31.8%.