TurboQuant llama.cpp CUDA Fork, Measured on an A100 — turbo4 Matches q4_0, turbo3 Breaks at Long Context

TurboQuant llama.cpp CUDA Fork, Measured on an A100 — turbo4 Matches q4_0, turbo3 Breaks at Long Context

The TurboQuant numbers circulating for llama.cpp come from Apple Silicon (Metal) and consumer RTX cards. We had a different question: what does the CUDA fork actually do on a datacenter GPU, on a model people serve, with the measurements people skip — perplexity *at long context*, decode speed *with a full cache*, and VRAM?

Setup: Qwen3-8B Q4_K_M, one A100 80GB, the spiritbuun CUDA fork built with -DGGML_CUDA=ON for sm80, flash attention on, six KV cache types (f16, q8_0, q4_0, turbo4, turbo3, turbo2) plus the two asymmetric K/V mixes the community recommends. Every number below is from this machine; the JSON logs are attached.

Four results:

1. turbo4 is a genuine q4_0 replacement — same perplexity (+0.65% vs f16), 27% smaller, and 2.5x faster decode than q8_0/q4_0 at 64K context on this kernel.
2. turbo3 is fine at 4K and broken by 16K. Perplexity goes from +5.5% at 4K to +24% at 8K to +210% at 16K. Don't ship it without checking your context length.
3. The fork's quantized-KV attention path is slow on A100 for q8_0/q4_0 — at 64K depth they decode at 29% of f16 speed. TurboQuant's kernels do better, f16 does best.
4. The asymmetric mix K q8_0 / V turbo3 is nearly lossless (+0.18%) but inherits q8_0's slow decode.

This is Part 5 of our TurboQuant series. Part 3 explains why this is a fork and not upstream; Part 6 reimplements the algorithm and explains *why* the numbers look this way.

1. What We Measured, and Why These Four Things

Most TurboQuant benchmarks report llama-bench prefill throughput and a perplexity at -c 512 or -c 4096. Those are the numbers that make KV compression look free, because they don't exercise the thing KV compression changes: reading a large quantized cache on every decode step.

So we measured:

• Perplexity on wikitext-2 at context 4,096 (40 chunks), and again at 8K / 16K / 32K to see how quality scales with the amount of quantized context each token attends over.
• Prefill throughput at 8K and 32K prompts (llama-bench -p).
• Decode throughput at depth — tokens/s generating with 8K, 32K and 64K tokens already in the cache (llama-bench -d). This is the only speed number that reflects KV bandwidth.
• Peak VRAM at a 32K context, sampled with nvidia-smi during a single-chunk perplexity run.

Model: Qwen3-8B (36 layers, 8 KV heads, head dim 128 → 144 KiB of f16 KV per token). Weights Q4_K_M (4.68 GiB).

2. Perplexity at 4K: turbo4 = q4_0, turbo3 costs 5%

!Perplexity change vs f16 KV

At 4K context the story is tidy. turbo4 lands exactly on q4_0 (8.373 vs 8.374) with fewer bits. turbo3 costs 5.5% — noticeably more than the "+1%" reported from Metal on Qwen3.5-35B, consistent with the pattern that smaller models are more sensitive. turbo2 is not usable. And keeping keys at q8_0 while dropping values to turbo3 is within noise of f16 — the K/V asymmetry the community discovered is real and large.

3. Perplexity vs Context Length: turbo3 Falls Apart

This is the measurement that changed our recommendation. Same model, same file, perplexity with the context window widened so each token attends over more quantized cache:

Each cell is perplexity over 3 wikitext-2 chunks at that context (change vs f16 in parentheses). Absolute values shift with chunk count; comparisons across types are on identical text.

turbo4 degrades gently. turbo3 does not: at 32K the perplexity is roughly three times the f16 value. Whatever error turbo3 introduces per key, it compounds with the number of keys each query has to rank, and at 3.25 bits the softmax stops being able to find the right ones. The community's "context-scaling regression" reports were about *speed* at long context on Metal; this is a *quality* regression on CUDA, and it is much worse.

The asymmetric K q8_0 / V turbo3 mix stays flat across context lengths — another sign the damage is on the key side. If you want 3-bit values, keep the keys at 8 bits.

Practical rule: turbo3 is a ≤4K-context setting on this fork — it already costs +24% at 8K. For anything longer, turbo4 or an asymmetric mix.

4. Speed: Prefill Is Cheap, Decode at Depth Is Where Types Diverge

!Prefill and decode throughput vs f16

Prefill (tokens/s, higher is better):

Prefill costs 10–17% for the turbo types on A100 — the Walsh–Hadamard rotation and codebook lookup on the store path are not free on this kernel, unlike the "2% faster than q8_0" Metal result. turbo3 decode from an empty context is already 39% slower than f16, which hints at the dequant cost per token.

Now the number that matters — decode with a full cache:

!Decode tokens/s at 8K, 32K, 64K depth

Two surprises. First, on this fork's CUDA build, q8_0 and q4_0 decode terribly at depth — at 64K they run at under 30% of f16. Their flash-attention path uses the non-tensor-core "vec" kernels for quantized KV, and at long context that dominates. Second, TurboQuant's kernels are the fastest quantized option: turbo4 at 64K is 2.5x faster than q8_0 and 2.7x faster than q4_0. The fork's fused dequant-in-attention path was clearly written with depth in mind.

But f16 is still fastest. On an 80GB card with an 8B model, you don't *need* to compress the cache for memory until well past 128K — and until you do, f16 is both the best quality and the best speed. KV compression on a big GPU is a capacity decision, not a speed one.

5. VRAM at 32K Context

!Peak VRAM at 32K context

Peak VRAM includes the 4.7 GB of Q4_K_M weights and ~0.4 GB of compute buffers; the differences are the KV cache. The analytic f16 KV for 32K tokens is 4.83 GB (36 layers × 8 heads × 128 × 2 × 2 bytes × 32,768), and the measured deltas track the nominal bits-per-value closely. turbo4 and q4_0 are the same size; the choice between them is speed and quality, both of which favor turbo4.

Where this matters: a 70B Q4_K_M model leaves ~34 GB free on an 80GB A100, and f16 KV at 128K context for a 70B (80 layers, 8 KV heads) is ~40 GB. That's the case where turbo4 turns "doesn't fit" into "fits with room for batch."

6. Recommendations (A100, this fork)

And one meta-recommendation: always run llama-perplexity at the context length you actually serve. A 4K perplexity number said turbo3 was fine. It isn't.

7. What We Did Not Measure

• Task accuracy (needle-in-a-haystack, GSM8K) — Part 4 does that in vLLM.
• Batch > 1. llama-bench decode is single-stream; batched serving shifts the bandwidth picture.
• Other GPUs. The q8_0/q4_0 depth penalty is specific to this fork's CUDA flash-attention path on sm80; upstream llama.cpp and Hopper may differ.
• Other models. Everything here is Qwen3-8B; Llama-3.1-8B or a 70B may shift the turbo3 threshold.

Reproduce: build the fork (cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=80), then bench-llamacpp-tq.sh, bench-llamacpp-depth.sh, ctxscale.sh, and kvmem-probe.sh from the attached bundle. Total wall-clock about 2.5 hours on one A100.

References

• spiritbuun/llama-cpp-turboquant-cuda — the fork benchmarked here
• llama.cpp Discussion #20969 — community results on Metal/RTX
• Part 3: TurboQuant Status Check · Part 6: TurboQuant From Scratch
• Zandieh et al., TurboQuant, ICLR 2026