Neural networks are kinda clean

Two recent posts disagree about whether neurons are readable. Nick Cammarata argues that most neurons have one main job and that weak activations usually don't matter. Leo Gao argues that clean-looking top examples can fool you, and proposes a test: replace the neuron with a simulation built from its explanation, and check whether the model still works.

I ran that test on 278 random MLP neurons of Qwen3-4B, and also ran the weak-activation ablation where the two posts split.

What I found

A few neurons are clean. About 4% of sampled neurons (11 of 278) are fully explained by one sentence; they're almost all single-token detectors in the first or last layers.
For most neurons, the story is true but not enough. Explanations correctly describe when a neuron fires strongly, but a simulation built from them is about as damaging as deleting the neuron (median −1% damage avoided on ordinary text).
Weak activations matter in language models. Zeroing each neuron's weakest 75% of activations changes the top next-token prediction on 59% → 30% of tokens from Qwen3 0.6B to 8B.

Weak-activation ablation results across Qwen3 model sizes — Zeroing each neuron's weakest activations, across Qwen3 sizes.

Per-neuron damage avoided by simulation — Damage avoided per neuron when it is replaced by a simulation of its explanation.

Caveats: automated explanations from about 60 examples per neuron, one model for the simulation test, and very small per-neuron effects. The full method, numbers and caveats are in the repository.