Original title: SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers
Shaowen Wang, Ge Zhang, Kairong Luo, Yuhao Wu +5 · September 2026
Standard AI models can grow deep and powerful, but making them larger takes huge amounts of computer power. Some models try looping, which means running data through the same middle layers more than once to act deeper. Past tests gave these looping models unfair extra computing time. To test this properly, researchers created SMELT. SMELT takes a Mixture-of-Experts model, which directs data through specialized mini-networks, and loops the middle half of its layers twice. Crucially, they strictly matched the compute, memory, and total parameters to a standard model. SMELT learned faster and used 6.8--18.0% less training compute. It scored better on tasks, especially Code, and improved on longer text. Looking inside, the second pass stops the model from wasting attention on useless tokens.
Imagine you're packing a suitcase for a long trip. Normally, you'd put everything in at once, but that might lead to a messy bag where you can't find items easily. Instead, you pack the essentials first, then go back and add the extras in a second pass, which helps you organize better without extra time. In this paper, the AI model does a first pass through its processing steps, then revisits the middle steps once more, which saves compute and improves how well it learns.
Training large AI models consumes massive computing resources, which costs money and energy. By saving compute, this work could make developing advanced models cheaper and more accessible for researchers and companies.
The abstract does not specify all limitations, but it notes the advantage is largest on code, so it may not benefit all tasks equally. The work is based on specific scaling laws and ablations, and it is unclear if it generalizes to other architectures or sizes without further study.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers — Shaowen Wang, Ge Zhang, Kairong Luo et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.