Hybrid Mamba-Transformer, Measured — Qwen3.5-9B Fits 4.4x More Context and 3.6x More Requests on the Same A100

Hybrid Mamba-Transformer, Measured — Qwen3.5-9B Fits 4.4x More Context and 3.6x More Requests on the Same A100

Our first post on hybrid Mamba-Transformer models argued from three papers that the 2026 convergence — three quarters linear layers, one quarter attention — buys a much smaller inference cache and faster long-context decoding. It cited "~25% of full KV cache" and "~2–3x inference speed" from the papers. This post checks those claims on hardware we control. The short version: Part 1 holds up — the cache lands at a measured 22% (claimed ~25%) and the speed advantage is real — and half of this post is about which measurements can legitimately show it.

Setup: Qwen3.5-9B (32 layers: 24 Gated DeltaNet + 8 full attention, full_attention_interval=4) against Qwen3-8B (36 layers, all attention), both bf16, one A100 80GB, batch 1, contexts from 2K to 64K tokens, HuggingFace Transformers 5.16 with the flash-linear-attention Triton kernels. We measure the cache — key/value tensors for attention layers, recurrent and convolution state for linear layers — by summing every tensor in the cache object, not by reading GPU memory (which would include logits).

1. The Cache: 4.4x Smaller at 64K, Approaching 4.6x

The numbers decompose exactly, which is the satisfying part.

Transformer: 36 layers × 8 KV heads × 128 head-dim × (K+V) × 2 bytes = 147 KB per token. 61,839 tokens → 8.49 GB. ✓

Hybrid: only the 8 attention layers grow with context. They use 4 KV heads × 256 head-dim × 2 × 2 bytes = 32 KB per token. The 24 Gated DeltaNet layers hold a fixed-size state regardless of context — a 32-head × 128 × 128 recurrent matrix plus a 4-wide convolution buffer per layer, about 25 MB total in bf16. So cache ≈ 25 MB + 32 KB × tokens. At 61,839 tokens: 1.94 GB. ✓

Two things follow. The asymptotic ratio is 147 / 32 = 4.6x, not the "4x" you'd get from the 8/32 layer count, because Qwen3.5 also uses fewer KV heads. And at short contexts the ratio is lower (2.4x at 2K) because the fixed linear state is a larger share — hybrids pay a small constant cost up front and win increasingly as context grows. The first post's "~25% of full KV cache" was, if anything, conservative.

For serving, this is the number that matters: at 64K context, one A100 holds roughly 30 concurrent Qwen3.5-9B sequences in the memory it takes to hold 7 Qwen3-8B sequences. That is the hybrid's actual product.

2. Speed: What We Measured, and Why It Doesn't Answer the Question

Here is the honest part. We also timed prefill and decode in HuggingFace eager mode:

Read naively, the hybrid loses on prefill and ties on decode. That reading is wrong, and it is worth understanding exactly why, because it is the same trap anyone benchmarking a new architecture in a research framework falls into:

• Decode at ~25 tok/s for both is the Python loop, not the model. An A100 decodes an 8B model at 130–150 tok/s in llama.cpp or vLLM; HuggingFace eager decode is bound by per-step framework overhead, which is identical for both architectures. The measurement has no resolution left to see a KV-bandwidth difference.
• Hybrid prefill is kernel-bound in a way the transformer isn't. The transformer's attention runs on PyTorch's fused SDPA kernels, tuned for years. The Gated DeltaNet layers run on flash-linear-attention's Triton kernels — and we could not build causal-conv1d against this machine's CUDA 12.1 toolchain, so the convolution falls back to a slow path. The first 2K-context run took 42 seconds, almost all of it Triton compilation. The hybrid's prefill throughput doubles from 8K to 64K (746 → 3,519 tok/s) while the transformer's falls (9,877 → 5,400), which is exactly the linear-vs-quadratic shape the theory predicts — but at 64K the hybrid is still at 65% of the transformer, because of kernel maturity, not architecture.

So: this experiment confirms the memory claim and cannot confirm or refute the speed claim. A fair speed comparison needs a serving engine with production kernels for both paths. So we ran one — vLLM 0.28, same GPU, same prompts — and the picture flips. See the next section.

A note on peak memory, since it surprised us: peak allocation during 64K prefill was higher for the hybrid (47.7 GB vs 41.7 GB). That is not the architecture either. Qwen3.5's vocabulary is 248K tokens versus Qwen3's 152K, and HuggingFace materializes logits for every position during prefill: 61,839 × 248,320 × 2 bytes ≈ 30 GB. A serving engine never does that.

3. The Follow-Up: Same Question, Production Kernels

We repeated the speed measurement in vLLM 0.28 (both models supported natively, bf16, same A100, same prompts):

With production kernels the hybrid wins everything, and the shape matches the theory: the advantage grows with context (decode at 4K is +11%, at 32K +16%; prefill goes from −51% in HF to +37% here). But note what the numbers do not show: the "2–3x inference speed" from the papers. At 32K context on an 8B/9B pair, the measured speedup is 16–37%. The dramatic multiplier lives in the capacity column — 3.6x more concurrent long-context requests per GPU — which turns into throughput only when your workload is memory-bound. That is the honest version of the hybrid pitch.

4. What Changed From the First Post

One correction to our own framing: the first post described the saving as coming from "75% linear layers." The larger lever in Qwen3.5-9B is that its 8 attention layers use 4 KV heads where Qwen3-8B uses 8. Half of the per-layer saving is GQA width, not the hybrid split. Architecture comparisons across model families always bundle several decisions; the cache arithmetic above is how to un-bundle them.

5. Reproduce

python -m venv ~/venvs/hybrid && ~/venvs/hybrid/bin/pip install torch==2.9.1 --index-url https://download.pytorch.org/whl/cu128
~/venvs/hybrid/bin/pip install "transformers>=5.9" flash-linear-attention
CUDA_VISIBLE_DEVICES=0 python bench-hybrid-hf.py --model Qwen/Qwen3-8B   --out hybrid-hf.json
CUDA_VISIBLE_DEVICES=0 python bench-hybrid-hf.py --model Qwen/Qwen3.5-9B --out hybrid-hf.json

The script sums numel × element_size over every tensor reachable from past_key_values — the only apples-to-apples cache measure when one model's cache is K/V tensors and the other's carries recurrent states.

New measurements go out weekly with Paper of the Week — subscribe below if you want the next one.

Harness: bench-hybrid-hf.py, bench-hybrid.py (attached).

Hardware: 1× A100 80GB, driver 535. Software: transformers 5.16 + flash-linear-attention 0.5.2 / vLLM 0.28.0; both models bf16.

Wall-clock: ~40 min (HF) + ~25 min (vLLM). Raw results: hybrid-hf.json, hybrid.json.

Verified on: 2026-08-31.

References

• Part 1: Hybrid Mamba-Transformer MoE — Three Teams, One Architecture
• Qwen3.5 model card and config (layer_types, full_attention_interval) on Hugging Face
• flash-linear-attention — Triton kernels for Gated DeltaNet
• vLLM PR #39931 — hybrid-model support used in the follow-up