Original title: Cliff: Learning Process Rewards from the First Mistake
Peixuan Han, Runhui Wang, Ketan Ramaneti, Jie Hao +2 · September 2026
When training large language models, reinforcement learning often only checks whether the final answer is right or wrong. This gives the model very little guidance on the steps in the middle. Other fixes can require specialized reward models or force the learner to copy a teacher's exact thinking style. To solve this, researchers built Cliff. The idea is simple: once a model makes its first mistake, checking the rest of its reasoning tells us very little, because the foundation is already ruined. Cliff uses a standard, ready-to-use language model as a teacher to spot where that first mistake happens in a response. It marks everything before the mistake as correct and everything after as incorrect, turning this split into detailed rewards. Across 12 different scenarios, Cliff consistently improved reasoning, even when using modest teachers.
Imagine you are learning a new recipe and follow it step by step. If you add too much salt in the first step, the dish will be ruined no matter what you do later, and just knowing the meal is bad isn't helpful. Here, the AI's reasoning process is like the recipe, and the method finds the exact step where the first error occurs to give targeted feedback. This way, the AI learns to avoid that specific mistake and improves its overall performance.
For anyone relying on AI that explains its reasoning, like in tutoring or problem-solving apps, this makes those explanations more dependable. It helps AI learn more effectively from each attempt, leading to better performance in tasks that require step-by-step thinking.
The abstract doesn't specify the diversity of the 12 test scenarios, so generalizability to all reasoning domains is uncertain. It requires an off-the-shelf LLM teacher, which might not be available or effective for all models or tasks.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Cliff: Learning Process Rewards from the First Mistake — Peixuan Han, Runhui Wang, Ketan Ramaneti et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.