Original title: Scaling Near-Optimal SFT-RL Annotation Budget Allocation from Small to Large LLMs
Jingtan Wang, Arun Verma, Xiaoqiang Lin, Zhengyuan Liu +3 · September 2026
Training large language models involves two main steps: supervised fine-tuning, which uses direct examples of good answers, and reinforcement learning, which scores answers to guide the model. Teaching models requires expensive human annotations, and researchers do not know the best way to split a fixed budget between these two steps. Instead of hunting for one perfect split, researchers searched for a near-optimal zone—a range of budget mixes that get within 2-10% of peak performance. They found this forgiving zone is wide, grows wider for bigger models, and transfers reliably from small test models to large target models. This means developers can test budget splits on cheap, small models, knowing the winning recipe will still work on giant models.
Imagine packing for a trip with a strict weight limit, and you have to split your allowance between clothes and books. Normally, you'd stress over finding the perfect balance, but it turns out that many different splits work almost equally well. If you pack a small bag first, you discover a wide range of acceptable splits, and this same range works even when you scale up to a larger suitcase. In this paper, the researchers find that for AI training, there's a similarly wide range of budget splits between two methods that perform nearly optimally, and this range expands with model size, so small experiments can guide large-scale decisions.
AI developers training large language models can save time and money by using small-scale experiments to guide resource allocation, avoiding costly large-scale searches.
The abstract doesn't state explicit limitations, but the findings are based on specific tasks, model families, and RL methods, so generalizability to all scenarios isn't guaranteed. It also doesn't address other resource constraints beyond annotation budgets.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Scaling Near-Optimal SFT-RL Annotation Budget Allocation from Small to Large LLMs — Jingtan Wang, Arun Verma, Xiaoqiang Lin et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.