Original title: Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR
Boyan Li, Bingsen Chen, Chenghao Yang, Ping Nie +2 · September 2026
Teaching AI models to solve hard math and logic problems usually relies on two techniques. One is learning from teacher hints on every word, called on-policy distillationA method where a student model learns from a teacher model by imitating its decisions during training.. The other is practicing through trial and error, called reinforcement learning. Past work tried mixing both methods together at the exact same time. The researchers tested doing them one after another instead: use the teacher hints first, and then let the model practice on its own. It works because the teacher first shows the model a wide variety of possible correct solutions. Then, solo practice trains the model to zero in on the best paths. Doing both at once causes the two signals to interfere. They found that running them in order beats combining them.
Imagine learning to cook a new dish. First, you watch a chef prepare it several times to understand the steps and ingredients. Then, you try cooking it yourself while the chef corrects your technique in real time. If you try to watch and cook simultaneously, you might mix up the order or measurements. In this paper, the two-stage approach is like separating the watching and cooking phases, so the model first learns the range of possibilities and then refines its actions without confusion.
This finding helps developers train more capable AI systems for tasks like math and logic reasoning without complex training setups. It simplifies the process, potentially leading to better performance with less effort.
The abstract doesn't say what specific limitations the work has, but it notes that the results are based on logic and math reasoning benchmarks, leaving generalizability to other domains unclear.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR — Boyan Li, Bingsen Chen, Chenghao Yang et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.