LLM Reasoning Failures Part 2: Cognitive Biases — Inherited from Human Data
Anchoring, Order Bias, Sycophancy, Confirmation Bias — cognitive biases from RLHF and training data, tested across 7 models.
LLM Reasoning Failures Part 2: Cognitive Biases — Inherited from Human Data
LLMs learn from human-generated text. The problem is, they inherit human biases along with it.
In Part 1 we examined structural limitations like the Reversal Curse, counting failures, and compositional reasoning breakdowns — fundamental architectural constraints that persist regardless of scale. This Part 2 focuses on robustness issues. The model doesn't necessarily give wrong answers; rather, its answers shift depending on how you ask the question.
Structural limitations cannot be fixed by scaling models up. But cognitive biases are different. They stem from biased training data and RLHF, so they are in principle improvable. The problem is that they are still observed across every model today.
We tested 4 cognitive biases across 7 models: GPT-4o, GPT-4o-mini, o3-mini, Claude Sonnet 4.5, Claude Haiku 4.5, Gemini 2.5 Flash, and Gemini 2.5 Flash-Lite.
1. Anchoring Bias
What Is Anchoring?
Anchoring is the tendency for judgments to gravitate toward an initially presented number. It is one of the most well-studied biases in human psychology. Real estate agents showing expensive listings first, negotiators throwing out the first number — these are all anchoring strategies.
Does the same phenomenon appear in LLMs?
Experiment Design
Related Posts
Hybrid Mamba-Transformer, Measured: Qwen3.5-9B's Cache Is 4.4x Smaller Than Qwen3-8B's — and Why Our Speed Numbers Don't Count
Cache memory of Qwen3.5-9B (24 linear + 8 attention layers) vs Qwen3-8B measured from 2K to 64K context on one A100: 4.4x smaller at 64K, converging to 4.6x, and it decomposes exactly. Plus why HF eager speed numbers can't judge the architecture.
Paper of the Week #1 — Sparse Is Not the Same as Interpretable
I trained BDH and counted more than 4.8 billion activations across three controls. Training moved its single-latent sparsity from 49.98% to 81.65%, but a similar-budget ReLU baseline reached 91.04%. Sparsity is real; by itself, it is not evidence of interpretability.
Reproducing Claude's Watermark Locally — SynthID-Text on an Open Model, With Detection and Removal Attacks
We applied SynthID-Text — the algorithm Claude adopted — to Gemma 2 2B and measured everything: why detection is impossible without the key, how many tokens it needs, why false positives explode on short text, and how one rewrite by a local 3B model erases the watermark.