How Much Does the Hint Matter? Scope Conditions for Hindsight Self-Distillation in Financial Reasoning

How Much Does the Hint Matter? Scope Conditions for Hindsight Self-Distillation in Financial Reasoning. By Johnny Yu and one other, from the University of California, Berkeley and CTGT. Published on the twenty-sixth of October.

Financial institutions working under data-residency constraints can only use open-weight models small enough to host locally, and those models trail frontier systems on multi-step quantitative reasoning. On-policy distillation can narrow that gap by supervising a student on its own rollouts. The traditional version needs a strong external teacher, though, and on-policy self-distillation, or OPSD, conditions on a full reference solution. Both ask a lot. One needs a bigger model, which is exactly what these institutions cannot run, and the other needs a worked answer for every problem. The question here is how much of that signal is actually necessary, and the answer the authors report is: much less than it seems. Their claim is that the gain comes from where supervision is applied, not from what the hint says.

The vehicle is Fin-SD, a localized hindsight self-distillation recipe. When the student fails a problem, the student itself finds the first incorrect reasoning step and writes a short critique of it. The trajectory is cut just before the error, aiming to keep a correct prefix. A frozen copy of the student is then conditioned on that prefix plus the hint, and this hint-conditioned teacher is distilled into the student over a short rollout under reverse KL. No external model contributes a gradient. The student is, in effect, teaching itself from the point where it first went wrong.

Applied to GPT-OSS-20B on a custom financial reasoning benchmark, where formula inputs must be derived from realistic scenarios through accounting conventions, three findings emerge. First, swapping the critique for a generic-feedback control matches the whole gain of Fin-SD over the base model, three point zero three percentage points of accuracy on the FinanceReasoning Hard subset. Second, the hint writer need not be stronger than the hint receiver. Hints from GPT-OSS-120B, a model of the same family at roughly six times the parameters, bring no materially greater benefit than self-written ones. Third, Fin-SD matches self-distillation from full reference solutions on accuracy, while generating nine point five percent fewer tokens.

The scope is narrow by design. One student model, one domain, and a custom benchmark of multi-step financial problems. Within that setting, the authors also check for collateral damage: general reasoning on MMLU-Pro and a fourteen-metric compliance audit on FinTrust show no material regression under Fin-SD. What the results speak to is this recipe at this scale, not distillation in general.

The title asks how much the hint matters, and in this setting the reply is not very much. A generic nudge placed at the first wrong step does as well as a careful critique, and a self-written hint does as well as one from a model six times larger. Location, rather than content, is carrying the signal. And for an institution that cannot call out to a frontier model, that is a useful thing to learn: the recipe needs nothing beyond the model it already hosts, and it spends fewer tokens than the reference-solution alternative to get there.

How Much Does the Hint Matter? Scope Conditions for Hindsight Self-Distillation in Financial Reasoning. By Johnny Yu and one other, from the University of California, Berkeley and CTGT. Published on the twenty-sixth of October. This is the long version: the whole paper rather than the abstract.

A bank that cannot let customer records leave its own infrastructure cannot call a frontier model through an API. It is confined to open-weight models small enough to host on its own hardware, and those models trail on exactly the multi-step quantitative reasoning financial analysis needs. Fine-tuning can close that gap, but domain fine-tuning is known to erode instruction-following and the other behaviours that made the model acceptable to deploy. So the institution needs a way to improve reasoning with no stronger external model in the loop, and without breaking compliance. The question the authors ask on the way is sharper than that: when a model teaches itself, how much does the teacher actually need to know?

Two ideas come first. On-policy distillation samples answers from the student and has a teacher score every token of them, so the student is corrected in the states it actually reaches rather than in a teacher's. On-policy self-distillation removes the outside teacher: the teacher is the student itself, shown privileged information the student does not see, which in prior work is a full verified reference solution. Because teacher and student share weights and differ only in their context, this is context distillation. The aim is to internalize whatever behaviour the extra context induces, so the model reproduces it with no context at inference. A third ingredient comes from HINT-SD, built for long-horizon agents: in a failed trajectory most steps were fine, so supervise only where the failure happened.

The recipe is called Fin-SD, and the student is GPT-OSS-20B. Each epoch begins by mining failures: the student attempts every training problem, and a judge compares its answer to the ground truth. On a failure, a frozen copy of the student is given the question, the failing trajectory, and the correct answer, and asked three things. Where is the first step that departs from a valid solution? What is a short critique of that step? And is the error arithmetic or conceptual? Models are poor at returning exact token positions, so it quotes the faulty clause verbatim and the quote is located in the text. Pure arithmetic slips get no gradient, since a calculator tool would fix them at deployment. An informal audit of that classifier found it right on forty-eight of fifty items.

The trajectory is then cut immediately before the error. Student and teacher both see the question plus that correct prefix, drawn from the student's own generation; only the teacher also sees the critique, after a preamble telling it to silently apply the correction and continue. The student rolls out just one hundred tokens from there, and at each position its next-token distribution is pulled toward the teacher's under reverse KL divergence, with the prefix masked out. At every epoch boundary the teacher is re-frozen from the current student and the failures are mined again, so the target improves as the student does. In practice that leaves roughly twenty-five problems per epoch: about sixty failures, of which around thirty-five are arithmetic.

Evaluation is the FinanceReasoning Hard subset, two hundred and thirty-eight problems, with five seeds per configuration and confidence intervals reported. Fin-SD lifts accuracy from sixty-nine point one six to seventy-two point one nine percent. Against self-distillation from full reference solutions it matches on accuracy, at seventy point four two percent for the baseline, while generating nine point five percent fewer tokens. Supervised fine-tuning on reference solutions actually loses about six point seven points.

The comparison that tests the thesis is the placebo. The critique is replaced by one fixed string, "Your mistake is conceptual", with everything else unchanged. It scores seventy-two point five two percent: no worse than the real hint, inside the seed spread. Swapping the hint writer for GPT-OSS-120B, roughly six times larger, gives exactly the same accuracy under reverse KL. Smaller ablations favour reverse over forward KL and the short rollout over a full one, by around one point each. Out of domain, MMLU-Pro moves by about half a point and none of the fourteen FinTrust compliance metrics shows a large regression.

The paper has a limitations section, and it is candid. The self-distillation ceiling is real, and the authors expect writer-invariance to hold only up to some difficulty, beyond which the student cannot diagnose its own mistakes. Seventeen of the two hundred and ninety-five seed formulas appear in solutions to twenty-two evaluation items, so this is in-domain transfer. Hyperparameters were barely tuned: warmup occupied fifteen of twenty total optimization steps. And two hundred and twenty problems is a modest corpus.

The evidence adds its own cautions. The full-solution baseline's interval over the base model includes zero, so matching it is a low bar, and Fin-SD's own lower bound sits barely above zero. The compliance and MMLU-Pro checks were single runs. HINT-SD found the opposite on hint writers, and the authors offer two untested explanations: a smaller four B student there, and feedback that supplied information the model lacked.

The idea to carry away is a scope condition. When a capable model already knows the convention and simply fails to pick it, the hint barely matters: what matters is stopping it at its first wrong step and making it try again from a clean prefix. That is a recipe in which no external model contributes a gradient, which is the point for an institution whose data cannot leave the building.