Learning to Learn from Context: Synthetic Training from Perturbed Public Documents

Learning to Learn from Context. Synthetic Training from Perturbed Public Documents. By Haoyi Wu and six others, from the AllSpark Team. Published on the twenty-seventh of September. No code repository is linked to it.

Many real tasks ask a large language model to learn from the material in front of it: complex context specific to that task, rather than knowledge already stored in its weights. Models remain weak at this. The obvious remedy, human annotation of such contexts, is expensive and hard to scale. Public documents of high quality are abundant, but there is a catch. Much of the public web was already consumed during pretraining, so training naively on those documents would reward the model for remembering them, not for learning from them. A model that already knows the answer has no reason to read the page, and a training signal built on that page would teach it nothing about reading. So the field is left with a skill it wants to teach, a data source that is too expensive, and a cheap data source that teaches the wrong thing.

The authors' way out is to use those public documents after perturbing them a little. Their empirical finding is that, given such lightly altered documents, language models can generate reasoning traces and answers that genuinely depend on the context. Those traces and answers then become training data for a student model. The perturbation is what breaks the shortcut: a model can no longer lean on a memorized version of the text.

The synthesis pipeline has four steps. First, it rewrites each source document to reduce the risk of memorization, so that what the model saw in pretraining is no longer quite what it is being shown. Second, it generates questions, along with grading rubrics, that require reasoning over the document rather than a quick lookup. Third, it answers those questions with the document supplied as context, which is where the reasoning traces come from. Fourth, it filters, admitting only the samples whose answers truly depend on the document. That last gate is the one that keeps memorized answers from slipping back in through the side door. No human annotators are involved at any point. From about three thousand five hundred documents, the pipeline produces about ten thousand samples.

Training on those samples substantially improves a student model on CL-bench. The student is Qwen3.6-35B-A3B, and it is trained in two stages. Supervised fine-tuning comes first, and on its own it lifts the student from thirteen point seven percent to twenty-two point eight percent.

A further reinforcement learning stage, rewarded against the rubrics, takes it to twenty-four point six percent. That puts it on a par with a frontier model of over a trillion parameters, Qwen3.8-2.4T, which scores twenty-three point nine percent on the same benchmark. The improvements also transfer broadly, to long-context understanding, to instruction following, and to reasoning. Code generation and knowledge, by contrast, stay mostly flat.

Where the gains land fits the aim. Abilities that lean on reading what is given move, while code and stored knowledge stay where they were, a pattern in line with a pipeline built to reward using the document rather than recalling it. The authors present the pipeline as a reproducible, scalable route to better context learning, and as a base for further research on reasoning that stays grounded in its context.