Learn AI by Building
From your first dataset to production agents โ deep-dive series, hands-on notebooks, and experiments you can rerun yourself.
Paper of the Week
All Issues โPaper of the Week #10 โ Reorganize the Agent's Workspace, and Haiku 4.5 Runs at a Quarter of the Cost
This week: EnvOpt (UC Berkeley, UChicago, Bespoke Labs) lets one agent rebuild the workspace another agent works in, without answer keys. On AutomationBench, Claude Haiku 4.5's cost fell 72.7% and its pass rate rose from 2.33% to 16.67%. The same workspace cut Opus 5's cost by 68.1%, with pass rate moving from 33.83% to 35.17%. We checked every number against the paper's tables and note where the shared figures need context. Also: SlackBench, from the same group, where the best of 33 agent setups scored 80.25% on questions about a research team's Slack, and reading the workspace as plain files beat Slack tools by 7.85 points on average.
Premium Series
Our Products
Courses and code built from what we measure here
Video Courses
13 hands-on courses โ quantization, diffusion, RAG, on-device AI, decision models. $199 lifetime bundle, first 3 lectures of every course free
LLM Quantization and Compression Hands-On
GPTQ, AWQ, GGUF, QLoRA โ fit LLMs into the memory you have. The course behind our KV-cache measurements
Free Benchmark Code
The scripts and raw results behind the measurement posts, starting with the KV cache harness. Free with an account
Premium Series
140 deep-dive posts across 21 series, bilingual KO/EN, with production-ready code and notebooks
Latest Posts
View All โ
MiniMax H3 Video + Audio on 8 GB VRAM? Inside FreeVideo's Adaptive Execution Engine
We ran FlashML-org/FreeVideo's live Adaptive Execution Planner across 9 hardware configs and aggregated 146 community benchmarks. On 8 GB VRAM + 16 GB RAM, 0 of 50 transformer blocks stay in VRAM and a single 10s 768p clip reads 76.1 GB from SSD plus 173 GB over PCIe; upgrading system RAM to 32 GB cuts SSD streaming to 0 GB.

90% of Your Agents Should Just Be Python Scripts: Anatomy of Quadratic Token Costs
We ran the same 10-file daily transaction triage (180 records, 3 anomalies) on Claude Haiku 4.5 two ways. A runtime agent loop burned 102,546 input tokens and 20.7s across 12 turns and failed on turn 12. Asking the same model once (379 input tokens) to write a Python script passed 30/30 runs in 38.8ms each โ 7,569x cheaper hourly even with 100% prompt caching.

Stopping Silent Failures in Production Agents: Don't Let LLMs Grade Their Own Homework
We tested 20 agent runs (12 silent failures, 8 valid passes) across self-report, two LLM-as-a-Judge setups on Claude Haiku 4.5, and a plain Python code gate. An LLM judge seeing only reports and test logs missed 7 of 12 failures; even with full git diffs it passed a 100+10=120 invoice. The Python gate caught 12/12 in 0.33 ms at zero tokens.

Connecting 8 MCP servers burned 47,800 tokens before our first prompt: measurements and a 91% diet
We measured 8 popular MCP servers (134 tools) with Anthropic's official count_tokens API and Claude Haiku 4.5. Notion alone took 24,441 tokens, all 8 took 47,800. Description compaction saved only 15.6%. An allowlist cut billed input tokens by 81.8% while lifting in-scope accuracy from 10/12 to 12/12; deferred loading cut total input tokens by 91.3% at the cost of a second round-trip.

Swapping vLLM for SGLang: the two-card answer held, but on one card 5 of 32 users waited in line
Same Qwen3.8-27B and two A100 PCIe cards, swapping vLLM for SGLang 0.5.23. Two-card winners held (split for 1 user, copies for 64), but on one card SGLang capped concurrency at 27, leaving 5 of 32 users in line with 14.4 s TTFT p95.

Where Decision Models Read Their Answer: laya, DiffusionGemma, Kev, CLM and a Plain Classifier in Code
Five ways to turn a customer message and 77 intent labels into a probability per label, followed through the code of each. On the same 154 BANKING77 messages they scored from 3.2% to 90.3%. laya cuts every label to three tokens, and giving it more room lifts it from 37.0% to 46.1%; the others keep labels whole, place the options differently, and read the score elsewhere.