Cliff: Learning Process Rewards from the First Mistake

Cliff. Learning Process Rewards from the First Mistake. By Peixuan Han and five others, from Amazon Web Services and the University of Illinois Urbana-Champaign. Published on the second of September. No code repository is linked to it.

Reinforcement learning with verifiable rewards has become a standard way to post-train language models, but it leans on a coarse outcome reward, so it says almost nothing about the intermediate reasoning that produced the answer. The existing fixes each add a constraint. Process reward modeling requires a specialized reward model. On-policy distillation assumes the teacher and student share reasoning patterns, which they often do not.

The observation the method is built on is simple and, once stated, hard to argue with. Once a reasoning process has first gone wrong, evaluating everything after that point adds very little, because all of it is already conditioned on an invalid prefix. You are grading careful work done on a false premise. So the useful piece of information in a failed rollout is not a score over the whole thing. It is the location of the first mistake.

Cliff uses an off-the-shelf language model as a teacher to identify that first mistake in each rollout. That single point naturally splits the rollout into two parts: a correct prefix and an incorrect suffix. The signal is then converted into token-level advantages, positive across the prefix and negative afterwards. No specialized reward model is trained, and nothing assumes the teacher reasons the way the student does, which removes both of the constraints the alternatives carry.

Across twelve different scenarios the method consistently improves reasoning performance, beating on-policy distillation by fifteen percent and standard GRPO by seven percent. The detail that matters most for adoption is that this holds even with teachers of modest capability, which follows from what the teacher is being asked to do: locating the first error in a chain is a much easier judgment than producing a better chain. The authors also analyse the role of ground truth in the method and examine its training dynamics.

The economy of it is the appeal. One judgment per rollout, from a model you already have, turned into dense token-level feedback, because knowing where reasoning broke tells you the sign of everything on both sides of that point. The expensive part of process supervision was never the judgment. It was building something qualified to make it at every step, and locating one boundary replaces that entirely.

Cliff. Learning Process Rewards from the First Mistake. By Peixuan Han and five others, from Amazon Web Services and the University of Illinois Urbana-Champaign. Published on the second of September. This is the long version: the whole paper rather than the abstract.

The question this paper asks is unusually well-posed, and it is worth stating in their words: how fine-grained does a process signal really need to be? Everyone agrees outcome rewards are too coarse. The assumption underneath the existing fixes is that the remedy is to get finer, ideally per step or per token. This work argues the remedy is one bit of structure per rollout, in the right place.

Reinforcement learning with verifiable rewards became standard because automatically verifiable outcomes make training scalable without human annotation. But the outcome reward evaluates only the final result, so it cannot distinguish reasoning of vastly different quality. A nearly complete solution containing one minor error is penalized identically to a completely incorrect attempt. The two established alternatives each add a constraint. Process reward models require training an additional reward model, which is less generalizable and susceptible to reward hacking. On-policy distillation reaches its best performance only when teacher and student share similar reasoning patterns and come from the same family.

So Cliff divides each rollout exactly once, at its first mistake, yielding two segments. A correct prefix and an incorrect suffix. A teacher model judges the student's rollouts and locates that boundary. For a rollout that fails to reach the answer, tokens before the boundary receive relatively higher advantages, while tokens after it receive negative feedback exactly as standard training would give. That is the whole mechanism, and three properties follow from its shape. It keeps the simple formulation of the surrounding method while supplying process-level feedback. It is model-agnostic and task-agnostic, because the supervision depends only on whether reasoning remains correct up to a given point, not on the domain or on how the teacher happens to reason. And it is naturally resistant to reward hacking, which is the property that deserves the most attention.

Before the training results, the authors establish the thing everything depends on: that language model teachers can locate the first problematic step accurately, even when the teacher's own performance on the task is imperfect. That asymmetry is the practical heart of the paper. Finding where an argument breaks is a much easier judgment than producing a better argument, so the teacher does not need to be better than the student at the task, only better at noticing. Across twelve different scenarios, the method consistently improves reasoning performance, beating on-policy distillation by fifteen percent and standard GRPO by seven, and it holds with teachers of modest capability. The authors also analyse the role of ground truth in the method and examine its training dynamics.

The limitations here are mostly implied rather than listed, which is itself worth registering: a paper whose central claim is that a coarse signal suffices carries an obligation to say where it stops sufficing, and this one largely does not. Everything depends on an evaluator capable of identifying the first mistake, which the authors state as a condition of applicability; where that judgment is unreliable, the signal is too. The formal-logic motivation is explicitly not a proof, and it should not be read as one, since a wrong step does not always make everything after it worthless in practice. And the comparison is against two specific alternatives rather than the full space of process supervision, so the comparison establishes that Cliff beats two specific approaches with known weaknesses rather than that finer supervision has nothing further to offer. The work was done during an internship at a cloud provider, which is worth knowing only because the teacher model is an off-the-shelf one and the economics of that assumption depend on what you already have access to.

What generalises is the economy. One judgment per rollout, from a model you already have, converted into dense token-level feedback, because knowing where reasoning broke fixes the sign of everything on both sides of that point. The expensive part of process supervision was never making the judgment. It was building something qualified to make it at every step, and locating a single boundary replaces that entirely.