Paper of the Week #1 — Sparse Is Not the Same as Interpretable

Paper of the Week #1 — Sparse Is Not the Same as Interpretable

A model can have mostly-zero activations and still tell you very little about why it worked. I trained the strongest counterexample—a brain-inspired recurrent architecture—against three controls. The result changes what I will demand when a paper calls a system interpretable.

Aug 17–26, 2026 · BDH-CQ · StateM · EnvHarness · Apodex 1.1

The Claim

For agents, interpretability is becoming a property of the system around the model, not of the weights inside it.

A sparse activation vector may be easier to inspect than a dense one. It is not, by itself, evidence that the unit means one thing, that a decision can be reconstructed, or that a failure can be repaired. Those are different claims. This week’s strongest systems make their state, transitions, and checks explicit outside the network. The strongest counterexample says the opposite: its recurrent, brain-inspired substrate is inherently interpretable because its activations are sparse and positive.

That gives the claim a fair test. If BDH’s sparsity survives an untrained control, a corpus control, and a comparable ordinary ReLU network, the internal story gets stronger. If it does not distinguish the architecture, “sparse” is a description—not a receipt.

The Receipt: BDH-CQ, and the baseline its sparsity was missing

Engdahl et al. (Pathway-led; Bielik AI and NYU) · Aug 10 · arXiv:2608.09888

BDH-CQ is the most interesting paper here, not a straw man. Its 150M-parameter model reaches 29.5% pass@2 on the public ARC-AGI-1 evaluation set at a computed $0.00070 per task. That is not frontier accuracy. It is a new point on the reported cost–accuracy frontier, and it comes from a genuinely different computation: demonstrations update a recurrent memory, then the query is solved by iterating in a high-dimensional latent space without narrating intermediate reasoning as tokens.

Its substrate is BDH, or Baby Dragon Hatchling (arXiv:2509.26507). BDH implements working memory through synaptic plasticity and Hebbian updates rather than a conventional KV cache. The original paper calls the architecture inherently interpretable and says its activation vectors are sparse and positive. It also presents separate monosemanticity analyses. One distinction matters: the paper’s abstract gives no sparsity percentage. So every sparsity figure below is something I measured, not something I am quoting.

BDH-CQ’s weights are not public, so I could not reproduce the ARC number. The baseline BDH implementation is public under MIT and small enough to train directly. I used that to test the narrower premise carrying the interpretability story: does its exact-zero rate tell us something distinctive about the architecture?

What I ran

I trained a 25.3M-parameter BDH on 1.1MB of Shakespeare for 1,200 steps, byte-level, on one A100. Its best validation loss was 1.5163 and its final loss was 1.5464. I then measured 4.83 billion activation values per BDH condition.

The measurement site must be named. Each layer produces x_sparse = relu(...) and y_sparse = relu(...), then multiplies them. Unless I explicitly say “product,” the numbers below refer to the first single post-ReLU latent, x_sparse.

After training, that latent was 81.65% exact zero. The product was 97.46%. Then came the missing cells.

Untrained control. At initialization, the single latent was 49.98% zero, but the product was already 75.10% zero. The other ReLU was 50.30% zero. If those two masks were independent, the predicted product-zero rate would be 1 − (1 − 0.4998)(1 − 0.5030) = 75.14%.

The measured value missed that prediction by 0.04 percentage points. In other words, three quarters of a 97.46% product sparsity figure is available before one gradient step. Training adds a real 22.36 points to the product, but quoting the endpoint alone would turn multiplication into evidence of learning.

Corpus control. I byte-matched a second corpus containing half code and half prose and repeated the run. The first latent landed at 81.50% zero; the product at 97.26%. I expected the small, repetitive Shakespeare corpus to be inflating sparsity. It was not. The difference in the single latent was 0.15 points, evidence in BDH’s favor and against my prior.

Architecture control. A conventional ReLU Transformer with a similar parameter budget (27.5M) trained on the same Shakespeare split produced 91.04% exact zeros in its FFN hidden activations—9.39 points above BDH’s first latent.

Here is the limitation before the conclusion: this was not a matched-quality comparison. The Transformer finished at 2.3798 validation loss, versus BDH’s 1.5464. A weaker model can carry more dead ReLU units, so some unknown portion of its 91.04% may be failure, not architecture. The activation sites are analogous, not identical; there was one seed per trained condition; and a 25M model on 1.1MB of text does not establish behavior at BDH-CQ’s 150M ARC operating point. I did not measure monosemanticity, causal faithfulness, or state interpretability.

The result therefore does not show that Transformers are more interpretable, nor that BDH is not monosemantic. It shows something narrower and useful: exact-zero percentage, measured this way, does not distinguish the supposedly interpretable architecture from an ordinary ReLU baseline.

Training clearly changes BDH: 49.98% becomes 81.65%, a 31.67-point rise in the single latent. The effect repeats across two very different corpora. That is a real architectural behavior worth explaining. It simply cannot carry the larger word “interpretability” alone. For that, I want a causal demonstration that a named state predicts and controls a named behavior—or an external record that lets another person reconstruct the decision.

That second kind of evidence is where the rest of this week points.

The Contrast

StateM — the operational version of the claim

Qin, Lu, Wang, Wang; work conducted in the authors’ personal time · Aug 15 · arXiv:2608.15089

StateM does not interpret neurons. It makes an agent run inspectable: durable states, phase-local context, checked transitions, recoverable runbooks, and versioned practices shared by human and agent. With no weight updates, GPT-5.5 xhigh moved from 83.1% to 92.1% on Terminal-Bench 2.1. GPT-5.6 Sol xhigh reached 95.3% raw accuracy across 445 trials, solving every one of the 89 tasks at least once.

That is the kind of interpretability I can operate. When a run fails, I can point to the state, the unmet precondition, and the transition that should have been blocked. The caution is equally concrete: on BusinessBench, held-out gains were only 0.55 macro and 1.34 micro points; the 10.04-point gain came from two mechanism-matched families. A runbook is a domain asset, not magic middleware.

EnvHarness — inspect the other side of the loop

Huang et al. (Google Cloud AI Research-led; UNC Chapel Hill and WashU) · Aug 20 · arXiv:2608.19880

EnvHarness moves the same idea from agent to environment. A Stage changes the starting state, a Contract rewrites actions or observations, and a Chain extends the task, while the original verifier remains intact. EnvRigger diagnoses a policy from its rollouts, writes a targeted wrapper, then accepts it only after fresh validation.

Across five benchmarks in four domains, skills learned through these environments improved held-out results by up to 9.0 points while using 9.8% fewer interaction steps. This is not evidence that an individual decision is semantically transparent. It is evidence that the learning intervention is inspectable: code names what changed, the old verifier still grades it, and a failed mutation leaves a trace. BDH asks us to inspect hidden state; EnvHarness asks us to inspect the contract around it.

Apodex 1.1 — direction, not independent evidence

Apodex Team · Aug 24, revised Aug 25 · arXiv:2608.23283

Apodex combines executable environments, coordination, and an AgentOS that preserves task state and provenance across tools and agents. Its 35B Mini makes the direction commercially relevant: capable local deployment plus long-horizon orchestration.

But the model, agent framework, benchmarks, and reporting come from the same team, and several evaluations are internal. The paper does show process traces detailed enough to inspect, but inspectability is not comparative superiority. I will use it as a map of where products are going, not as evidence for this issue’s claim.

Where I’d Be Wrong

Two observable results would change my view.

First, a matched-loss rerun in which BDH retains high sparsity while the ReLU Transformer falls near 50% would show that my control was dominated by dead units; I will score that after training both models to validation loss 1.60 or lower on the same split. Second, a causal intervention study on the open BDH—identify units from the authors’ monosemantic procedure, ablate them, and predict the affected concepts in advance—would make internal state more operational than the external audit trail I currently prefer. I will grade the first in issue #2 and the second when weights and a runnable protocol permit it.

Ship It · Code of the Week

Use github.com/henryqin1997/statem, the authors’ Apache-2.0, Python 3.11+ command-line state machine for long-running agents.

Monday morning, do one thing: take the last ten agent failures in your repository and add one executable exit check for the most common failure. Not another paragraph in the system prompt—a command, predicate, checklist, or approval that blocks the transition.

For example, replace “verify the article before publishing” with a verify state that cannot advance until links resolve, the chart asset exists, and every bold numerical claim appears in the source ledger. StateM stores the current node, check results, evidence, and history outside model context. That gives you the property a sparse vector does not: another person can see exactly why the run moved forward.

The Ledger

There is no previous claim or correction to score in issue #1. The ledger starts by locking the measurement rule: the primary number is the first single post-ReLU latent, x_sparse; the product must always be labeled separately. That makes this issue’s 81.65% comparable with the matched-loss run promised next week.

Next issue scores the promised matched-loss control and applies a checked runbook to a real publishing task. A week counts when the experiment runs, including a null result or one that weakens my claim.

Reproduction streak: 1 week

The companion notebook contains the experiment. The measurement script and raw JSON are retained with this issue in the repository.

References

• BDH-CQ: In-Context Learning with Recurrent Latent Reasoning — arXiv:2608.09888
• The Dragon Hatchling — arXiv:2509.26507
• StateM: Reaching 95.3% Raw Accuracy, or a $15 Frontier Run — arXiv:2608.15089
• EnvHarness: Awakening Static Worlds for Agent Learning — arXiv:2608.19880
• Apodex 1.1: Scaling Agentic Intelligence for Complex Work — arXiv:2608.23283