Gemma Already Ran the Ablation

A paper from Yann LeCun's group dropped this week: The Spike, the Sparse and the Sink (arXiv:2603.05498). It makes a clean claim about two famous transformer oddities — and our lab spent one afternoon checking whether that claim holds on the exact models we work with. It does, and the way it holds settled a real decision for us. Here are the numbers.

The claim

Two quirks show up in almost every modern language model. Massive activations: a handful of channels carry values in the thousands while everything around them stays small. Attention sinks: the model dumps a huge share of its attention onto the first token, no matter what that token says. Both have been treated as curiosities for years.

The paper shows they are one story. The standard Llama-style recipe — pre-norm residual stream plus RMSNorm — manufactures the spike as a kind of built-in bias vector. Normalization then flattens that spike into a near-constant direction, and that constant direction becomes the geometric target the attention sink locks onto. Most importantly: the authors suppress either phenomenon with architectural tweaks (QK-norm, sandwich norm, long-context training) at no measurable cost to model quality. These are artifacts of the recipe, not features of intelligence.

Why we cared

Papers study their models. Your rig runs yours. We are selecting a host model for a steering-research project — work that writes low-magnitude signals into a model's middle layers. If those layers already carry a built-in spike a thousand times larger than your signal, you are whispering into a jet engine. So the question wasn't academic: are our candidate checkpoints spiky, and exactly where?

We ran a census. Three models, pure inference, one consumer GPU, one afternoon: Qwen2.5-1.5B, Gemma-4-12B, and Qwen3-14B. The Gemma sits size-wise between the two Qwens on purpose — if the phenomenon tracked scale, the 12B should land between its neighbors.

The result

Both Qwens spike hard — and the bigger Qwen spikes harder. The Gemma, sitting between them in size, carries nothing: peak-to-median ratio of 1.98, a gentle rise and taper, no step-up block, no wall. Architecture, not scale, settled in three data points.

The mechanism, found in the source

Why is Gemma clean? We read its modeling code. Gemma-4-12B ships the paper's entire suppressor catalogue in production: QK-norm (the paper's strongest lever, ~99.9% spike reduction in their ablations), sandwich normalization on every sublayer's output (~86% lever), value normalization, attention softcapping. Every knob the paper measured — already turned. Qwen uses the standard pre-norm-only recipe and keeps the textbook spike. For a beautifully illustrated tour of Gemma-4's architecture — including the normalization stack that does the suppressing — Maarten Grootendorst's visual guide is the best on the internet.

Google quietly shipped the paper's control condition. Gemma didn't avoid the ablation study — it already ran it.

One caveat the paper predicted and we confirmed: the sink survives. Gemma still parks ~49% of its attention mass on position 0. Spikes and sinks are separable — Gemma killed one and kept the other. Practical consequence for anyone reading attention maps or activation norms as signals: mask position 0, always, on every model. Otherwise your metric measures the architecture, not the behavior.

What broke along the way (kept in, on purpose)

Two things went wrong, and both are the honest part of the story. First: our initial spike detector used a peak-to-median heuristic — which is blind to a spike that never comes down. Qwen2.5's spike persists across nearly the whole stack, so peak equals median and the ratio reads "normal." The third model broke the instrument, the instrument got fixed to flag absolute magnitudes, all results were recomputed. The third data point earned its keep.

Second: before measuring, we pre-registered a bold hypothesis about our own pipeline's relationship to these spikes. The measurement battery falsified it cleanly. That stings for about a minute and then it's just better knowledge — a pipeline component we thought might be an architecture correction turns out to be justified on entirely separate grounds. We pre-register so that our hits mean something; the price is that the misses go on the record too.

The takeaway

If you steer, probe, fine-tune, or monitor language models: census your host first. Half a day, pure inference, no training. You will learn which layers carry built-in walls, whether your instruments need position-0 masking, and — as it did for us — the census may flip a model-selection decision entirely: one of our candidate hosts had target layers sitting inside its 13,000-magnitude spike band. The other's were clean floor. Without the census, we would not have known which whisper reaches the room and which one drowns.

The geometry you write into is not a given. It's a property. Measure it.