A tiny subset of neural network updates controls most behavioral changes.

When researchers fine-tune AI models, they often use Low-Rank Adaptation, or LoRA. This method updates models by adding small, low-rank adjustments called writes. But it does not show which specific parts of those writes truly drive behavior. To find out, the researchers created Learned-Basis LoRA. They start by warming up a normal adapter. Next, they turn its learned write directions into a tidy, fixed geometric basis. Then, they continue training only within those fixed directions. They discovered that the truly useful changes are remarkably sparse and structured. The real behavior concentrates into very few directions, far fewer than standard settings suggest. In tests, using just two or four top directions per module hit optimal performance in all tested cases. Removing individual directions also showed that a tiny set of late-stage components carries an outsized impact on how the model behaves.

Why it matters

This matters for AI developers who fine-tune models, as it shows they can achieve the same results with far fewer changes, reducing computation costs and making model adaptation more efficient.

Questions answered