Original title: The Structure of Quantization Damage in LLMs: Why the Next Bit Should Be Spent Globally
Jundong Hu, Shekar Ramachandran · September 2026
Compressing large language models to save computing costs is called quantizationReducing the precision of numbers in an AI model to make it run faster and use less memory.. This process shrinks the precision of numbers, but it damages the model's accuracy. Researchers wanted to know where this damage happens and how to spend extra precision to fix it. To find out, they raised each layer back to 8-bit one at a time to see how much accuracy returned. They tested three common ideas about what causes the damage: task circuits, where the model computes, and weight statistics. None of these ideas predicted which layers actually benefited from restored precision. The damage was diffuse. For 8 of 9 models, fixing 75% of the lost accuracy required restoring roughly half the layers. Given an extra precision budget, spreading it globally across the whole model using finer quantization granularityThe level of detail in how precision is applied, such as per layer or per group of weights. worked much better than repairing specific sensitive layers, outperforming local fixes by 21-52 points.
You're organizing a library where books are sorted by category to save time. If you only rearrange the most important categories, the whole system stays inefficient. In this paper, the model's parts are like shelves, and quantization is like compressing book information; the study shows that fine-tuning the compression method across all shelves works better than optimizing specific shelves.
For anyone running AI models, quantization saves computing costs but can hurt performance. This research shows that spreading precision improvements across all layers is more effective than patching a few critical ones, making cheaper AI deployment more reliable.
The study is limited to post-training quantization on 9 open-weight models and specific quantization methods like RTN, GPTQ, and AWQ. It may not generalize to all quantization budgets, non-open-weight models, or architectures across families, as peak recovery location correlates with architecture within families but not across.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
The Structure of Quantization Damage in LLMs: Why the Next Bit Should Be Spent Globally — Jundong Hu, Shekar Ramachandran
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.